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PREFACE 


This study originated from requests from several airline training departments for help in analyzing 
the effectiveness of LOFT debriefings. Doug Daniel and Steve Gregorich helped identify crucial issues 
and ways to study these issues. 

The study could not have been conducted without the generous willingness of instructors and line 
crews to allow us to observe their debriefings. We are impressed with their high standards of 
professionalism. Training department managers from each of the airlines that participated in the study 
provided a wealth of background information and made valuable suggestions on early drafts of this 
manuscript. 

This study was funded by the FAA's Office of the Chief Scientist and Technical Advisor for 
Human Factors (AAR- 100). Eleana Edens, the program manager, provided support, encouragement 
and helpful suggestions. 
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LOFT DEBRIEFINGS: AN ANALYSIS OF INSTRUCTOR TECHNIQUES AND 

CREW PARTICIPATION 


R. Key Dismukes 1 , Kimberly K. Jobe 2 , and Lori K. McDonnell 2 


SUMMARY 


This study analyzes techniques instructors use to facilitate crew analysis and evaluation of their 
LOFT performance. A rating instrument called the Debriefing Assessment Battery (DAB) was 
developed which enables raters to reliably assess instructor facilitation techniques and characterize 
crew participation. Thirty-six debriefing sessions conducted at five U.S. airlines were analyzed to 
determine the nature of instructor facilitation and crew participation. Ratings obtained using the DAB 
corresponded closely with descriptive measures of instructor and crew performance. The data provide 
empirical evidence that facilitation can be an effective tool for increasing the depth of crew participation 
and self-analysis of CRM performance. Instructor facilitation skill varied dramatically, suggesting a 
need for more concrete hands-on training in facilitation techniques. Crews were responsive but fell 
short of actively leading their own debriefings. Ways to improve debriefing effectiveness are 
suggested. 


1.0 OVERVIEW 


How much crews learn in Line-Oriented Flight Training (LOFT) and take back to the line depends 
on the effectiveness of the debriefing that follows the LOFT. The Crew Resource Management (CRM) 
literature and the Federal Aviation Administration's (FAA) advisory circular (AC) 120-35C 
recommend that in the debriefing instructors should facilitate self-discovery and self-critique by the 
crew rather than lecture on what they did right and wrong. Self discovery by the crew is believed to 
provide deeper learning and better retention. Also, crews are more likely to enhance their performance 
of CRM in line operations if they develop their ability to analyze flight operations in terms of CRM and 
debrief themselves after line flights. 

In this study 36 LOFT debriefings conducted at five major U.S. airlines were analyzed. Audiotape 
recordings of each session were made with the permission of instructors and crews. The recordings 
were subsequently deidentified, coded, and analyzed for more than 70 variables. The Debriefing 
Assessment Battery was developed to systematically characterize instructor effectiveness at facilitation 
and the nature of crew participation in debriefings. The data indicate that the Debriefing Assessment 
Battery is a reliable and valid instrument for assessing instructors’ skill in facilitation and for analyzing 
crew participation. The battery was designed to be used by researchers, however a short form of the 
battery that can be used by training departments to evaluate debriefings in real time is currently being 
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developed and evaluated. 

Most instructors at all five airlines followed a similar general format for debriefing. However, 
within each airline both instructors and crews varied widely on many of the specific variables 
observed. There were also substantial differences among airlines on several variables for both 
instructors and crews, though most of these differences were not statistically significant due to the 
large variability within each airline. 

The debriefings lasted an average of 3 1 minutes, with a range of 8 to 82 minutes. However, 3 1 
minutes may not allow adequate time for crews to analyze their performance thoroughly or learn and 
practice the skills of self-debriefing. This study provides no data on the optimal length for debriefings, 
however an hour may be a useful rough target, with adjustments for the needs of individual crews. 
This suggestion must, of course, be considered in the context of other demands on instructors' time. 

Most instructors appropriately emphasized crew performance in the LOFT and achieved a balance 
between CRM and technical issues, although the range of instructor scores on these variables was very 
large. Instructors typically emphasized the things crews did well, but said little about things done not 
so well and spent little time suggesting ways to improve. Likewise, crews’ discussions of their 
performance tended to be factual descriptions of events and crew actions, with limited evaluation of 
performance or discussion of ways to improve. 

The content of the debriefings was driven almost exclusively by the instructors; crew members 
rarely brought up topics on their own initiative. Also, discussions revolved around the instructor, even 
when the instructor succeeded in getting the crew to do most of the talking: there was little back-and- 
forth discussion directly between crew members. The data indicate that crews were responsive but not 
very proactive. This may be in part because few of the instmctors explicitly told crews they should 
take a proactive role and perform their own analysis without depending on the instructor to lead them 
step by step. It may also be that instmctors themselves either do not fully accept or understand the 
concept of crews taking initiative and responsibility for the content of the debriefing. 

On average, instructors asked a large number of questions to elicit crew participation, directing 
their questions evenly among crew members. Participation by captains and first officers was quite 
similar. Participation by flight engineers (in three-person crews) was lower, but this difference was 
marginally significant. 

Most instructors appeared to be highly competent and conscientious in the traditional roles of 
instmctors, and most attempted to facilitate crew participation to some degree; however, their success 
in facilitation ranged from very good to poor. Instmctors who were effective in facilitation tended to 
use a combination of techniques, such as careful phrasing of questions to encourage crew self- 
analysis, strategic silence, active listening, and follow-up on crew-initiated topics. Probably more 
important than the use of any particular technique is the instructor's underlying focus on encouraging 
the crew to analyze for themselves the situations that confronted them in the LOFT and how well they 
managed those situations. 

Many instmctors unwittingly did things counterproductive to their own attempts to facilitate crew 
participation. In addition to failing to explicitly state expectations for crew participation and allowing 
the discussions to revolve around themselves instead of encouraging crew interaction, some 
instmctors failed to allow crew members enough time to formulate thoughtful responses to questions. 
Also, some instmctors engaged in long monologues, gave their own evaluations before eliciting crew 
self-evaluation, failed to push the crew to go beyond superficial description of their actions, and/or 
failed to encourage crews to analyze why things went well when they did. 

The wide range of instructor effectiveness in facilitation indicates that the airlines face an issue of 
standardization of this aspect of debriefing. The distribution of facilitation scores was distinctly 
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bimodal, with one group of instructors scoring in the good to very good range and another group of 
instructors scoring in the marginal range. Also, instructors who did well in one aspect of facilitation 
typically did well in all aspects (except stating expectations for crew participation), and those who did 
poorly in one aspect tended to do poorly in all aspects. These data suggest instructors' ability to use 
various techniques is determined at least in part at the conceptual level: Do they grasp the underlying 
concept of facilitation? Do they accept the concept? Is facilitation the type of approach for which they 
have ability? 

The CRM literature states that debriefings should be led by the crews themselves, using the 
instructor as a resource. Our data suggest that this goal, although worthwhile, is rather idealistic. 
Instructors become discouraged when, after a brief and rather abstract course in facilitation, they 
attempt to facilitate debriefings and discover that crews often do not immediately respond. We suggest 
that it would be more effective to teach instructors that facilitation should be adapted to the level at 
which the particular crew is able to respond. Facilitation can be conducted at levels ranging from high, 
which approaches the ideal of the debriefing being led by the crew, to low, in which the instructor 
leads the crew substantially, but in all cases debriefings should emphasize as much self-discovery by 
the crew as possible. 

Instructors are encouraged to attempt to facilitate at the highest level possible for a particular crew. 
Realistically, however, most crews do not yet have the skills and motivation needed to lead their own 
debriefings without substantial assistance from the instructor. It may be possible to change this 
situation over time if LOFT instructors consistently encourage crews to take a proactive role in 
debriefing their own training. 

Instructors sometimes mistakenly assume that using facilitation requires giving up their role as 
teachers in the debriefing. On the contrary, good facilitation in no way precludes the instructor from 
adding his or her own perspective to the discussion or from teaching specific points about CRM and 
technical issues as appropriate. Effective facilitators can integrate their teaching points into a group 
discussion in which the crew members are full participants. 

The study provides empirical evidence that facilitation can be used to substantially increase crew 
self-discovery and the depth of crew participation. Instructors, however, need additional training in 
facilitation. Facilitation training should emphasize hands-on practice in which instructors encounter the 
kinds of obstacles they are likely to face in actual debriefings. Initial training should be followed by 
mentoring by senior instructors who are themselves expert facilitators. A training manual that provides 
detailed suggestions for how to facilitate debriefings is forthcoming as a companion to this technical 
report. 


2.0 INTRODUCTION 


2.1 Background 

Line Operational Simulation (LOS) is widely used to provide opportunities for crews to practice 
CRM concepts in realistic and challenging simulated flight situations. As indicated in the FAA's AC 
120-35C (1995), LOS includes LOFT, Line Operational Evaluation (LOE), and Special Purpose 
Operational Training (SPOT). LOFT is the original "non-jeopardy" form of simulation training in 
which crews are not graded on their performance. Like LOFT, SPOT is used for training rather than 
evaluative purposes. In LOE crews are graded, which is required in those airlines that participate in the 
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FAA's Advanced Qualification Program (AQP). Both LOFT and LOE are full-mission simulations that 
include all phases of flight, whereas SPOT may be full-mission or only a segment of a flight tailored to 
focus on a particular training point. 3 

How much crews learn in LOFT and take back to the line depends on the effectiveness of the 
debriefing that follows the LOFT (Helmreich & Foushee, 1993). The simulation itself is a busy, 
intense experience, and thoughtful discussion afterward is necessary for the crew to sort out and 
interpret what happened and why. Instructors are expected to lead debriefings in a way that encourages 
crew members to analyze their LOFT performance for themselves. Rather than lecturing to the crew on 
what they did right and wrong, the instructor is expected to facilitate self-discovery and self-critique by 
the crew (Butler, 1993; Hawkins, 1987; Smith, 1994). 

CRM and LOFT programs have developed considerably since their inception almost twenty years 
ago. The concepts and the value of CRM are now generally accepted by both airline managers and 
pilots. However, it is not clear whether crews consistently think about and practice CRM in line 
operations (see discussion in Helmreich & Foushee, 1993). AQP is bringing to fore the issue of how 
well crews are actually able to practice CRM, because poor CRM can cause crews to fail a LOE 
(Bimbach & Longridge, 1993; FAA, 1991). In order for LOE programs to be effective and accepted, 
pilots must believe they are being graded on performance dimensions they understand and by criteria 
that seem appropriate and achievable. The ability of crews to analyze and evaluate their own 
performance in LOFT may predict their acceptance of LOE grading. 

2.2 What is Facilitation and Why Use It? 

The FAA's AC 120-35C on Line Operational Simulations (1995) describes the general concept of 
facilitated debriefings: 

The facilitator should not handle the debrief in a "teacher tell" manner but, instead, operate as a 
resource to crew members by highlighting different portions of the LOS that may be suitable for 
review, critique, and discussion. The discussion should be led by the crew themselves, using the 
facilitator and the videotape as resources for use during their critique... Self-criticism and self- 
examination are almost always present in these situations, and in many cases they are much more 
effective than facilitator criticism... Thus, the facilitator should do everything possible to foster this 
sort of self-analysis, while at the same time keep the debrief at a constructive level. In the role of 
moderator, the facilitator can guide the discussion to areas that he or she has noted.. .However, 
unless absolutely necessary, the facilitator should avoid "lectures" about what is right and wrong. 

The concept of facilitated debriefings appears to have been part of the early inception of LOFT 
(Lauber & Foushee, 1981). The origin of this concept is not clear, but it appears to have been derived 
from the use of facilitation in other business settings, such as retreats in which managers discuss their 
organizational goals and issues (e.g., Gibb, 1982; Mills & Roberts, 1981). 

The primary rationale for facilitating rather than lecturing is that crews can learn and remember 
much more when they participate actively and make their own analyses than when they listen passively 


3 This study examined only LOFT debriefings. Many of the findings, however, are also relevant to SPOT. At the time of 
this writing, LOE is just starting to come into use, and it is not clear whether airlines will consider LOE a tool for both 
training and evaluation or just for evaluation. Thus, the role of debriefings in LOE is not yet determined. This report 
addresses only the training objectives of LOS debriefings, not the evaluative aspect found in LOE. For consistency, 
LOFT is referred to in this report, although the findings are expected to genaralize to the training objectives of all forms 
of LOS. 
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to the instructor (Duvall & Wicklund, 1972; Smith, 1994). Another potential benefit of crew-centered 
LOFT debriefings is that they can help crews develop the habits of analyzing their own CRM 
performance on the line and conducting their own crew debriefings following line operations (Butler, 
1993). In practice, crew debriefings on the line in civil operations are as yet rare, although military 
crews often debrief their missions. Thus, the LOFT debriefing is an important tool for showing crews 
how to debrief and for illustrating the benefits of self-debriefing. 

Continental Airlines' (1992) handbook on LOFT facilitation techniques outlines a useful hierarchy 
of facilitation based on the concepts of discovery and ownership. According to this handbook, the goal 
of facilitation is to have crews recognize what they did well and what they need to improve 
(discovery), and to have crews make a commitment to continue or begin using desired behaviors and 
stop using undesirable ones (ownership). At the top of the hierarchy is "they see it, they say it." This 
is the ideal in which crews recognize and analyze their own performance. In the middle is "you help 
them see it, they say it." If crews are not able to recognize what they did well and what they can 
improve, the facilitator can lead them to self-analysis through questioning. Finally, at the bottom of the 
hierarchy is "you help them see it, you help them say it." When crews are unable to recognize or 
analyze their performance the facilitator must evaluate for them to ensure that they understand what 
went well or poorly, and why. 

A literature search conducted as part of this study revealed no studies that analyzed the specific 
needs and issues of LOFT debriefings in order to adapt the general concept of facilitation to this 
specialized setting, which differs substantially from most business settings. The training departments 
of many airlines provide their instructors written guidelines; however, these guidelines tend to be 
rather sketchy and most do not provide a detailed exposition of how to use facilitation. 

The general literature on facilitation in settings other than LOFT is also rather sketchy. This is a 
trade literature rather than a scientific literature, and very little empirical evidence is provided to support 
assertions, validate specific techniques, or qualify the range of settings in which advocated techniques 
may be effective. However, the general concept of facilitation has considerable face validity as a way 
to encourage self-discovery by crew members. Both the adult learning literature and the cognitive 
research literature suggest that self-discovery improves learning, retention, and the ability to apply 
knowledge in diverse settings. 

According to the facilitation literature, adult learning is typically self-directed (Cornwell, 1979). In 
general, adults dislike long lectures, they learn best from discussions with peers, they need to integrate 
new knowledge with what they already know as professionals, they want to be told up front what is 
expected of them, and their self-esteem is directly affected by classroom discussion (Zemke & Zemke, 
1981). 

Active participation requires crew members to process information more deeply than listening 
passively to an instructor's critique does (see, for example, Slamecka & Graf, 1978). Deeper 
processing leads to elaboration of the information in memory and enables better retrieval from memoiy 
when it is needed (Baddeley, 1990). 

Facilitation can help individuals develop problem solving and critical thinking skills (Gow & 
Kember, 1993). Research in several areas of expertise suggests that individuals are better at solving 
problems and applying their knowledge in diverse situations if they have a good metacognitive 
perspective of their technical skills (see Metcalfe & Shimamura, 1994). Metacognition refers to 
knowledge of one's own thought processes and the ability to keep track of what one is doing while 
analyzing problems and managing tasks. Debriefings that emphasize self-analysis and self-discovery 
help crews develop metacognitive skills for managing cockpit situations. One could argue that the 
concept of metacognition is implicit in the philosophy of CRM; for example, CRM teaches crews to 
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establish priorities and keep track of how they are managing their priorities during abnormal line 
situations. 

2.3 Techniques for Facilitation 

Most of the techniques for facilitating group participation that are suggested in the literature 
concern the use of introductions, active listening, questions, and silence. The use of video recordings 
to enhance discussion is also discussed. 

2.3.1 Introductions. An explicit introduction is necessary to clarify the role of the facilitator 
and the nature of the participation expected of the group (Casey, Roberts, & Salaman, 1992; Nelson- 
Jones, 1992; Gibb, 1982). A good introduction can also motivate the group to participate by providing 
a rationale for the session. 

2.3.2 Active listening. Good listening skills enable the facilitator to work with what the 
participants are saying and to encourage further participation. Active listening shows that the facilitator 
is attending to the speaker, understands what is being said, and wants to hear more. Active listening 
can range from a simple "uh-huh" or "okay" to echoing or reflecting in one's own words what a 
speaker is trying to communicate. 

2.3.3 Questions. According to the Socratic method, learning is facilitated by questioning, 
encouraging exploration, and pushing for explanation; not by lecturing and telling the answers (Casey 
et al., 1992). “Can you give me a specific example?” “How did you and the other person actually 
behave?” and “What were your thoughts in the situation?” are examples of questions that can aid self- 
assessment (Nelson-Jones, 1992). Mills and Roberts (1981) assert that, ideally, questions should be 
brief; open (i.e., non-restrictive, don’t imply opinion or judgment); and begin with who, where, and 
when for factual responses or what, how, and why for more in-depth and detailed answers. 

The use of probing questions encourages active and in-depth participation. Probing questions that 
ask participants to explain and justify their responses have been reported to be particularly effective 
(Jacobsen, Eggen, & Kauchak, 1989). Mills and Roberts (1981) identified seven types of probes that 
encourage continued participation: non-verbal (e.g., a nod); short verbal ("Uh, huh?"); "W" words 
(especially what, how, and why); statements such as "Tell me more."; echoing of participant words; 
reflection of what the participant said with different words but the same meaning; and specialized 
reflections that imply more than stated by the participant. (Also, see Eitington, 1986.) 

2.3.4 Silence. Sometimes group participants do not respond immediately to a leader's question. 
Most people find silence in a group setting uncomfortable, and leaders often allow no more than a one 
second pause before rephrasing a question or answering it for the group. However, one second may 
not be long enough for participants to formulate a thoughtful response. Studies show that waiting three 
to four seconds substantially improves both the number and quality of responses (Rowe, 1986; 
Jacobsen et al., 1989). The longer pause elicits longer, more confident responses from the group, as 
well as more numerous voluntary observations, participant interactions, and participant questions. 
Furthermore, responses from slower participants increase, speculative responses and evidence- 
inference statements increase, and failures to respond decline (Omstein, 1990; Rowe, 1974). 

2.3.5 Videos. Most airlines videotape the LOFT. Although the use of video is not a facilitation 
technique per se, it can aid facilitation. Instructors select segments of the videotape to show during the 
debriefing to help the crew observe and discuss their performance. The video can help the crew view 
their performance from a third-party perspective (FAA, 1995); it may also help the crew remember 
what happened. 

The literature cited above provides examples of facilitation techniques and a rationale for using 
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them, but unfortunately provides little in the way of detailed, practical guidance for using these 
techniques in particular group settings and integrating the techniques into the overall management of a 
session. In order for these techniques to be used effectively in LOFT debriefings, they must be 
adapted to the particular characteristics and demands of these debriefings. 

2.4 Research Questions 

Although the concept of facilitated debriefings is widely espoused in the CRM literature, little 
empirical research has examined what actually happens in debriefings. This study attempts to answer 
five major questions: 

1) To what extent do instructors attempt to facilitate crew participation and self-discovery in LOFT 

debriefings? 

2) What techniques do instructors use to facilitate and how effective are these techniques? 

3) Is facilitation a viable approach to encouraging crew participation and self-discovery? 

4) What is the character of crew participation, especially in terms of analyzing and evaluating their 

own performance? 

5) How much variation occurs among instructors and among airlines in the conduct of debriefings? 


3.0 METHODS 


3.1 Participants 

Thirty-nine LOFT debriefings conducted at five major U.S. airlines between June 1994 and May 
1995 were observed. All five airlines are large, well-established national companies; four are 
passenger airlines and one is a cargo company. At each of the airlines the first author observed four to 
eleven debriefings. (At the first company visited, a second research observer was also present at the 
debriefings and interviews.) The training department managers who arranged the observations were 
asked not to preselect which instructors and crews would be observed; rather, the selection was driven 
by the schedules of who was instructing during the three to five days each airline was visited. The 
observed debriefings represented all or most of the fleets operated by each airline, and at least one 
LOFT simulation of each scenario flown in each fleet was observed. Generally, one debriefing was 
observed per instructor and crew; however, four of the instructors were observed debriefing a second 
crew for the purpose of comparison. 

Permission to attend the debriefing and to audio tape the session was obtained from each instructor 
and each crew member, and assurance was provided that all data collected would be completely 
deidentified to assure anonymity for all participants. 

3.2 Procedures 

Prior to observation of the debriefings, the written scenarios for each LOFT were reviewed and 
managers in the CRM departments were interviewed. After each debriefing the instructor was 
interviewed and asked to rate the crew's CRM performance and technical performance on separate 
five-point Likert scales ranging from poor (1) to exemplary (5). Instructors were also asked for 
comments about the debriefing process. 
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The audio recordings of 36 of the 39 debriefings were transcribed into text in their entirety and all 
references to individuals and organizations were deleted. (Two of the recordings were not sufficiently 
intelligible for transcribing and the tape recorder failed during another debriefing.) Of the 36 
debriefings that were transcribed, 25 were from two-person crews, and eleven were from three-person 
crews (Table 1). 

3.3 Measures 

3.3.1 Descriptive measures. Each instructor and crew utterance was coded for nine factors 
and the coding was checked during data entry. (The factors and the coding rules are described in 
Appendix A.) From these nine factors 72 utterance variables were calculated (see Appendix B). Data 
were also extracted on the instructors' use of videotapes to illustrate the crews' performance in the 
LOFT, including the number of video segments played for crew discussion, the length of the segments 
played, and the extent to which the segments were discussed. The above data will be referred to as 
"descriptive" to distinguish them from the data generated using the Debriefing Assessment Battery 
described below. 

3.3.2 Debriefing Assessment Battery. The Debriefing Assessment Battery was developed 
to systematically characterize instructor effectiveness at facilitation and the nature of crew participation 
in debriefings (Appendix C). This battery provides subjective rating scales on several dimensions, 
with appropriate anchoring (Appendix D), and can be used by raters who have experience in CRM. 
McDonnell (1995) provides a detailed description of the development and validation of the battery. The 
battery was based on the adult learning and facilitation literature, existing rating scales by M. M. 
Connors (1995) and R. H. Moos (1994), face valid assumptions of what constitutes good facilitation, 
and the airline industry's guidance to their instructors on how to facilitate LOFT debriefings. The 
battery incorporates a seven-point Likert scale ranging from poor (1) to outstanding (7). 

The battery contains 28 items grouped into seven composite categories consisting of four items 
each. Five of the categories rate the instructor while the remaining two rate the crew. The five 
instructor categories are Introduction (letting the crew members know what is expected). Questions (to 
focus on topics and elicit crew participation). Encouragement (the degree to which the instructor 
encourages and enables the crew to participate actively and deeply). Focus on Crew Analysis & 
Evaluation 4 (getting the crew to analyze and evaluate their own performance), and Use of Videos (to 
remind the crew of what happened in the LOFT and provide a springboard for discussion). The video 
is not part of facilitation per se but its use is an important part of the overall structure of the debriefing. 
Items in the two crew categories — Crew Analysis & Evaluation and Depth of Crew Activity — were 
designed to correspond closely with items in the instructor categories. 

Two of the authors independently rated the instructors and crews from each of the debriefing 
sessions after listening to each LOFT session audio tape while reading the verbatim transcript. For 
each of the first 10 debriefings, the ratings on the individual battery items were compared and 
discussed before rating the next debriefing. During each discussion, if either believed any ratings 
needed to be changed based on issues raised by the other, the scores were revised accordingly, 
although no effort was made to reach consensus on each item. For the remaining 26 debriefings, 
ratings were not systematically discussed. 

Interrater reliability was determined by calculating Pearson correlation coefficients for the two 
raters' initial scores for each of the seven battery categories before discussion or any revision of 


4 This variable will be referred to as Focus throughout the rest of this report 
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scores. Pearson interrater reliability coefficients ranged from .73 to .91 for the seven categories of the 
battery (Table 2). 

Aside from reliability coefficients, data from the battery are based on the average of the two raters' 
scores for each item. Composite scores for each of the five instructor and two crew categories were 
calculated by averaging the scores for the four items in each category. 

3.4 Statistical Analyses 

Differences among airlines were examined by one-way analysis of variance (ANOVA). In cases in 
which the ANOVA showed significant differences among the group of airlines, a Bonferroni post-hoc 
test was used to determine which airlines differed significantly from the others. Differences between 
two and three-person crews were examined by a t-test. Differences between crew members (captain, 
first officer, and flight engineer) were examined by a Wilcoxon matched-pairs test. Statistical 
calculations were based on the full set of 36 debriefings, unless otherwise stated in the tables. For all 
tests significance was computed by the two-tailed method, using an alpha of .05. Spearman rank- 
correlation coefficients were calculated for all pairs of variables. Correlation coefficients are referred to 
as "statistically significant" if p < .05. These findings should be interpreted cautiously, however, 
because a large number of correlations were run and five percent of these can be expected to represent 
type I error 5 at the .05 alpha level. 

Four instructors conducted two debriefings; thus, each of these four instructors received two 
measurements for each of the variables associated with their performance. These two measurements 
were averaged to obtain a single data point (n = 32) for (i) calculation of means and standard 
deviations, and (ii) the analysis described below. The means with duplicate instructors' scores 
averaged (n = 32) are reported for scores on the Debriefing Assessment Battery. However, since 
differences between the two methods of calculating the means were minor for the descriptive variables, 
these means are reported for the full data set (n = 36). 

Data from these four instructors were used to explore the question of whether the large variability 
observed among instructors reflected stable differences among the instructors. Five variables were 
selected for this analysis: session duration, percent of group words uttered by the instructor, percent of 
instructor words addressing CRM, percent of instructor words addressing crew performance, and 
instructor scores on a composite QEF variable created by combining the Questions, Encouragement, 
and Focus categories of the assessment battery. For each of these variables the difference between the 
values for the two debriefings given by the same instmctors was obtained, providing a delta score. 

The average of the delta scores for these four instructors was compared to delta scores obtained by 448 
random pairings among instructors who gave only one briefing. 


4.0 RESULTS 


4.1 General Observations 

At all five airlines most debriefings were not conducted immediately after the LOFT. Instead, after 
a short break, the instructor and crew first returned to the simulator to conduct about two hours of 


5 Type I error represents the chance that differences will be assumed to be significant when they are not. 
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"batting practice" as rehearsal for the proficiency check that would follow the next day. A few 
instructors, apparently on their own initiative when scheduling allowed, reversed the order so they 
could debrief the LOFT before batting practice. 

At all airlines most debriefings followed the same general format. The instructor would either give 
a very short introduction or no introduction at all, and then lead discussion of segments of the LOFT in 
the chronological order in which they occurred. Rarely did the instructor engage the crew in setting an 
agenda for discussion, although some instructors invited general comments on the LOFT before 
starting the discussion of specific segments. In the four airlines with video equipment, the instructor 
generally used a video segment to begin the discussion of related portions of the LOFT. A few 
instructors varied this general format; for example, one instructor systematically went through the 
CRM categories displayed on a wall poster, asking the crew to identify places in the LOFT in which 
they had employed each category. 

For most variables large differences occurred among debriefings within each airline. For some 
variables substantial differences also occurred in the average values between airlines, although in most 
cases the within-airline variability prevented the differences between airlines from being statistically 
significant. 

4.2 Descriptive Data 

The average duration of the debriefings was 30.7 minutes (Table 3), with a range of 8 to 82 
minutes. Duration was negatively correlated with instructors' ratings of crews' CRM performance (r = 
-.49, p < .01) and technical performance (r = -.39, p < .05) and positively correlated with the 
proportion of instructors' words directed to negative aspects of crew performance or ways to improve 
(r = .51, p < .01) 6 . This suggests that instructors spend somewhat more time with crews that had 
more problems. 

Across airlines, instructors' ratings of crew performance averaged 3.6 (SD = .90) for CRM and 
3.5 (SD = .89) for technical on a 1 to 5 scale in which 1 = poor, 3 = average, and 5 = exemplary. No 
statistically significant differences were found among airlines. 

4.2.1 Participation. With two-person crews instructors (IPs) did an average of 61% of the 
talking, captains (CAs) 21%, and first officers (FOs) 18% (Table 4). Instructors participated 
significantly more than any of the crew members and the difference in participation between captains 
and first officers, though small, was also statistically significant. With three-person crews instructors 
did 49% of the talking, captains 20%, first officers 19%, and flight engineers (FEs) 13%. As with 
two-person crews, the amount of participation by instructors was significantly greater than any of the 
crew members. Though there were no significant differences in participation between captains and first 
officers in the three-person crews, the difference between first officers and flight engineers was 
statistically significant. While the percentage of participation was much higher for instructors than for 
crew members on average, the percentage of participation varied substantially among instructors; for 
example, the percentage of talking by instructors with two-person crews ranged from 35 to 85%. 

The percentage of the talking done by instructors was negatively correlated (p < .01) with the 
percentage of the talking done by each category of crew member (CA: r = -.62; FO; r = -.83; FE: r = 
-.77). In contrast, the percentage of talking by captains was not significantly correlated with the 
percentage of talking by first officers or flight engineers, but the percentage of talking by first officers 
was positively correlated with the percentage of talking by flight engineers (r = .68, p < .05). 


6 Appendix D lists all correlations we examined among variables, including those not shown in tables. 
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4.2.2 Content of discussion. The average percentage of words directed to CRM topics by 
instructors varied from 19 to 64 among the five airlines (Table 5). The percentage directed to CRM by 
crews varied from 25 to 68. The average percentage of crew discussion directed to CRM mirrored the 
percentage of instructor discussion directed to CRM at each airline. At most of the airlines, CRM 
topics occupied substantially more of the discussion than did technical topics. 

On average, 41% of instructor words and 52% of crew words were directed to the performance of 
the crew in the LOFT (Table 6). Instructors emphasized positive aspects of crew performance (18%) 
over negative aspects (3%) and ways to improve performance (4%). Most of the crews' words 
concerning performance were neutral descriptions of what they did (33%), compared to positive 
aspects (8%), negative aspects (6%), and ways to improve (5%). 

The content of the crews' remarks mirrored the content of the instructors' remarks. The 
percentages of crew words directed to discussion of CRM, technical, positive performance, negative 
performance, and ways to improve performance were all significantly positively correlated with the 
percentages of instructor words directed to these topics (Tables 7a and 7b). 

4.2.3 Instructor questions. Most instructors asked a large number of questions, averaging 48 
per hour among two-person crews (Table 8a). Among two-person crews, 60% of these questions 
were directed to specific crew members. Similar results were observed with three-person crews (Table 
9a). No significant differences were found in either the proportion of questions directed to each crew 
member or in the proportion of non-directed questions answered by each crew member (Tables 8b & 
9b), although the proportion answered by the flight engineer was substantially lower, falling just short 
of statistical significance (p < .06). 

4.2.4 Interruptions. Instructors frequently interrupted crew comments. The average number of 
interruptions per hour by instructors was 26 (SD = 16). (Active listening interjections were not 
counted as interruptions. See Appendix A for coding rules.) Twenty-one percent (SD = 13%) of all 
crew utterances (excluding S statements, defined below) were interrupted by the instructors, and 12% 
(SD = 8.7%) of all crew utterances were interrupted and never completed. No statistically significant 
differences in these variables were found among the airlines. Neither variable — percent utterances 
interrupted nor percent utterances interrupted and not completed — was significantly correlated with 
descriptive measures of crew participation (percent crew participation, number of crew analyzing 
utterances per hour, number of crew words per response, and number of crew SI words/hour) or 
crew variables measured by the Debriefing Assessment Battery. 

4.2.5 Videos. On average, instructors showed 8.8 (SD = 5.0) video segments per hour, each 
averaging 150 (SD =113) seconds in duration. No significant differences were found among airlines. 

4.2.6 Crew participation. Crew utterances were categorized as questions (Q); responses to 
instructor or crew questions (R); statements that add content to the discussion (SI); or other statements 
(S), most of which were concerned with maintenance of discourse (e.g., "I see what you mean"). 
Responses accounted for 44% of all crew words and S 1 statements accounted for 45% (Table 10). 

The distribution of the number of utterances among these four categories differed from the distribution 
of number of words because S statements were typically much shorter than the other three categories. 
The pattern of distributions among categories was similar among airlines. 

On average, individual crew members asked about six questions per hour. To analyze the character 
of crew questions, the set of all crew questions from airlines Y and Z (n = 98) were divided into three 
categories. Proactive questions address the content of the debriefing, raising new issues or bringing 
new information into the discussion (e.g., Did you realize I had not finished the checklist?). Reactive 
questions respond to a prompt without adding new information, usually to disambiguate what was 
said or meant (e.g., Do you mean the taxi checklist or the predeparture checklist?). Miscellaneous 


11 



questions are generally extraneous (e.g., "Do I have time for a coke?") or meta-conversational (e.g., 
"You know what I mean?"). 

Thirty-five percent of crew questions were proactive, 34% were reactive, and 30% were 
miscellaneous (Table 1 1). Sixty percent of the proactive questions addressed CRM, technical, or 
mixed topics, but only 12% of the reactive questions, and 7% of the miscellaneous questions 
addressed CRM, technical, or mixed topics. 

A few significant differences occurred among airlines in the number of proactive questions asked, 
but at all five airlines the number of proactive questions by crew members was small (Table 12). No 
significant differences were found in the number of proactive questions asked by captains, first 
officers, and flight engineers. 

Three other measures of crew participation were also examined: the number of analyzing utterances 
per hour, the number of words per utterance, and the number of words per response to the instructor's 
questions (Table 13). Analyzing utterances were defined as those that go beyond simple description of 
events and actions to examine underlying factors and how those factors influenced the outcome (see 
coding rules in Appendix A). The number of analyzing utterances per hour averaged 6.2 (SD = 4.7), 
with no significant differences among airlines or among the three crew member positions. The number 
of words per utterance and the number of words per response averaged 22 (SD = 10) and 30 (SD = 
17), respectively, with no significant differences among airlines or among the crew member positions. 

In general, discussion in the debriefings revolved around the instructor, even when the instructor 
got the crew to do most of the talking. Direct back-and-forth discussion between crew members was 
infrequent. To explore this aspect quantitatively, sequences of utterances by crew members were 
examined (Figure 1). Debriefings were analyzed in terms of blocks of crew utterances, each block 
beginning with the first crew utterance after an instructor utterance and continuing until the instructor 
spoke again. These blocks were mostly very short; 80% of them consisted of only one utterance by a 
crew member before the instructor spoke again; thus, in these blocks there was no crew interaction at 
all. Only 5% of the blocks contained four or more utterances by crew members. 

4.3 Debriefing Assessment Battery 

4.3.1 Scores. Average scores for instructor Questions, Encouragement, Focus, and Use of 
Videos and for crew Analysis & Evaluation and Depth of Activity fell close to 4, or adequate (Table 
14). Scores for instructor Introduction were much lower, averaging 1.6, which falls between poor and 
marginal. No significant differences were found among airlines in any category. 

The instructors' battery scores on use of Questions, Encouragement, and Focus were distinctly 
bimodal, with one mode peaking around 2 (marginal) and the other between 5 ( good) and 6 (very 
good). Table 15 and Figure 3 show this data for the five airlines combined. The separate data for four 
of the five airlines showed the same general bimodal pattern. In contrast, airline Y scores were all 
distributed around the higher mode and showed substantially less variance than did the scores of the 
other four airlines on these three variables. Scores for the two categories of crew participation at each 
airline also showed bimodal distributions similar to the distributions of instructor scores. 

4.3.2 Correlations. Crew scores on Analysis & Evaluation and Depth of Activity were 
significantly positively correlated with instructor Questions, Encouragement, and Focus, with 
coefficients ranging from .51 to .78 (Table 16 and Figure 2). Instructor Introduction and Use of 
Videos were not significantly correlated with crew scores on the battery. However, the third item in 
the Introduction category was significantly positively correlated with Crew Analysis & Evaluation (r = 
.45, p < .006), and the third item in the Use of Videos category was significantly positively correlated 
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with Crew Analysis & Evaluation (r = .45, p < .02) and fell just short of significant positive 
correlation with Depth of Activity (r = .38, p < .055). 

The five instructor categories were significantly positively intercorrelated with each other (Table 
17). In particular, use of Questions, Encouragement, and Focus were highly intercorrelated. The two 
crew categories were also significantly positively intercorrelated (r = .87, p < .01). 

4.3.3 Effect of introductions. The ten debriefings for which the instructor Introduction 
scores were 1.0 (the lowest possible score) and the nine debriefings for which the Introduction scores 
were the highest (ranging from 1.8 to 4.9) were analyzed further. Crew Analysis & Evaluation scores 
for the latter group were significantly higher than for the former group (Table 18). No significant 
difference between the two groups was found for Depth of Activity. 

4.4 Correlations Between Battery and Descriptive Variables 

4.4.1 Instructor battery with instructor descriptive. The correlations between the five 
instructor battery variables and seven instructor descriptive variables pertaining to how the instructor 
conducted the debriefing were examined (Table 19). The Introduction category was significantly 
positively correlated with number of directed questions, total number of questions, and percent of 
instructor words addressing CRM. The Questions category was significantly positively correlated with 
number of directed questions, total number of questions, and percent of instructor words addressing 
CRM and was significantly negatively correlated with percent participation by instructor and instructor 
words per utterance. Encouragement and Focus showed a pattern of correlation similar to that of 
Questions. Use of Videos was significantly positively correlated with percent of instructor words 
addressing CRM. 

4.4.2 Instructor battery with crew descriptive. The correlations between the five 
instructor battery variables and seven crew descriptive variables involving the nature of crew 
participation were examined (Table 20). The Introduction category was significantly positively 
correlated with crew words per utterance, words per response, and percent CRM. Encouragement was 
significantly positively correlated with crew percent participation, words per utterance, words per 
response, self-initiated words, analyzing utterances, and percent CRM. Questions and Focus showed 
a pattern of correlations similar to that of Encouragement, except that the correlations with words per 
response and self-initiated words were smaller and not statistically significant. The Use of Videos 
category was significantly positively correlated with percent CRM only. 

4.4.3 Crew battery with crew descriptive. Table 21 displays the correlations between the 
two crew battery categories and the seven crew descriptive variables. Both Analysis & Evaluation and 
Depth of Activity were significantly positively correlated with all seven descriptive variables except 
proactive questions. 

4.4.4 Instructor descriptive with crew battery and descriptive. The correlations 
between six instructor descriptive variables and a number of crew descriptive and battery variables 
were examined (Table 22). The percent of all speakers' words uttered by the instructor (i.e., percent 
instructor participation) was significantly negatively correlated with the crew variables: percent 
participation 7 , words per utterance, SI statements, analyzing utterances, proactive questions. Depth of 
Activity, and Analysis & Evaluation. Instructor words per utterance showed the same pattern of 
negative correlations, except there was no significant correlation with crew words per utterance. 
Number of directed questions per hour was significantly positively correlated only with percent of 


7 Forced correlation; see discussion 
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crew words addressing performance, and number of non-directed questions was not significantly 
correlated with any of these crew variables. The percent of instructor words addressing performance 
was significantly positively correlated with percent of crew words addressing performance and 
significantly negatively correlated with crew proactive questions. The percent of instructor words 
addressing CRM was significantly positively correlated with crew words per utterance, words per 
response, percent of crew words addressing CRM, and Crew Analysis & Evaluation. For most 
variables with which a significant correlation occurred for the crew as a whole, significant correlations 
also occurred for each crew member position separately (Appendix E lists the intercorrelations among 
all variables). 

4.5 Instructor Differences 

The delta score is a measure of how much two debriefings differ on a given variable. The delta 
scores for the four instructors who gave two debriefings were not significantly different from the delta 
scores for randomly-paired instructors for duration, percent CRM, or percent performance (Table 23). 
Instructor scores on the battery's Questions, Encouragement, and Focus categories were combined to 
create a QEF variable. For the QEF variable, the delta score of instructors who gave two debriefings 
was 34% of the delta score of randomly-paired instructors (t = -4.14, p < .005). 


5.0 DISCUSSION 


The five companies studied appear to be representative of large, well-established U.S. airlines. 
Although some differences occur, debriefings at these five companies show many common patterns. 
These findings, however, may not be representative of smaller, regional, or newly-started airlines, 
some of which have not developed CRM and LOFT programs to the extent that major airlines have. 

The large variability observed among instructors at each airline has important implications. For 
some variables the average values differed substantially among some of the airlines, although given the 
large variability, few of these differences were statistically significant. At airlines W and X, only four 
and five debriefings, respectively, were observed because not many LOFT sessions were run during 
our visits. With this small sample size and the variance observed, the standard errors for some of the 
mean values are large; thus, especially for these two airlines, the representativeness of these mean 
values is uncertain. 

For the reasons discussed above, one cannot conclude from these data whether real differences 
exist among the airlines on most dimensions (one major exception is emphasis on CRM, discussed 
below). What is clear is that individual instructors at each airline differed enormously in their 
effectiveness as facilitators and in their emphasis on CRM topics and crew participation. This large 
variability within all five airlines overshadows any differences that might exist among the airlines. This 
finding reveals an urgent need for additional training and standardization within each airline (see 
section 5.4). 

Some of the apparent variability among instructors may actually be within-instructor variability. 

For three descriptive variables that might seem characteristic of an instructor's approach — duration of 
debriefing, percent participation by instructor, and percent instructor words directed to CRM — as 
much variability was found between the two sessions given by the same instmctor as between 
randomly-paired sessions given by different instructors. These results should be interpreted with great 
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caution because of the small sample size (only four instructors conducted two debriefings), but they 
suggest that individual instructors may vary on these dimensions as a function of crew performance, 
external constraints on time, or unidentified factors. In contrast to the descriptive variable results, a 
direct measure of facilitation (combined scores for Questions, Encouragement, and Focus) showed 
much less variability between sessions given by the same instructor. Thus facilitation effectiveness 
may be a fairly consistent characteristic of the individual instructor. 

On several occasions crew members spontaneously volunteered that they had trouble remembering 
relevant aspects from the LOFT. The common practice of delaying the debriefing two hours or more 
until after the batting practice may have contributed to this memory difficulty. Performing the batting 
practice maneuvers, in the same cab as the LOFT and under similar conditions, is likely to interfere 
with the memory of the preceding LOFT. Unfortunately, we have no data addressing how much this 
practice interferes with the crews' memory, but we suspect it is not trivial and suggest that the issue be 
studied empirically. 

No industry standards exist with which to compare our observations on descriptive variables such 
as duration of sessions, percent discussion devoted to CRM and crew performance, how much of the 
talking is done by the instructor, etc. However, we discuss these variables below in terms of our own 
subjective impressions of how consistent the observed values are with objectives stated in the airlines' 
internal publications and with guidelines such as AC 120-35C (Line Operational Simulations). 

5.1 Descriptive Variables 

5.1.1 Duration. Most debriefings were fairly short: 3 1 minutes on average, including time 
spent watching videos (typically about 1/3 of the total session was spent watching video segments). It 
was clear that a half-hour session allowed the group to discuss only a few examples of the crew's 
performance, and often did not provide adequate time for in-depth analysis. Given all that occurs in a 
typical LOFT lasting over two hours and the importance of deep analysis of what happened and how 
the crew managed the situations confronting them, it seems highly desirable to spend more than 3 1 
minutes on debriefing. Although these data do not indicate what duration would be optimal, a 
thorough discussion was often accomplished in debriefings lasting about an hour. Instructors do need 
to vary the length of the session according to the training needs of the crew, but the 10-fold range of 
duration observed in this study is clearly problematic. 

Instructors who rated the crews' LOFT performance as high tended to conduct shorter debriefings. 
During interviews with instructors after each debriefing, some instructors indicated that some of them 
feel there is less to discuss with a crew that has performed well, and these instructors wanted to avoid 
"nit-picking" good performance. We suspect this attitude may shortchange high performing crews. It 
is important for these crews to analyze why things went well in order to help them make explicit the 
factors and behaviors that led to success. These behaviors may have been intuitive and may have 
depended on the compatibility of the particular two or three crew members involved. In order to take 
the lessons learned back to the line and apply then in situations in which the crew may not be so 
compatible, it would be helpful for the crew members to explicitly discuss what makes certain 
behaviors effective. Also, even high-performing crews need a chance to practice the as yet infrequently 
used skill of self-debriefing. 

5.1.2 Content. Substantial, statistically significant differences occurred among the airlines in 
the percent of discussion devoted to CRM, which may reflect differences in company training 
philosophy. At all but one of the five airlines, CRM topics occupied more of the discussion than 
technical topics. This emphasis is appropriate to the goals of LOFT. Very large differences also 
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occurred among instructors within each airline; at one airline, for example, CRM ranged from 6 to 
75% of instructor words. It is not clear whether these differences reflect different attitudes among the 
instructors toward CRM or indicate that individual instructors spend more time on technical topics 
when they perceived a crew to be deficient in technical knowledge or skills. However, the fact that the 
instructors' relative emphasis on technical topics was not correlated with their ratings of the crews' 
technical performance argues against the latter interpretation, or at least suggests that it is not the 
dominant factor. Regardless, a debriefing in which CRM topics plus mixed (CRM and technical 
combined) topics occupy less than a third of the discussion seems inappropriate. 

Discussion of the crews' LOFT performance was appropriately emphasized in the debriefings, 
accounting for roughly half of instructor and crew words, on average. This figure was fairly 
consistent across airlines. A good part of the instructors’ comments on performance were positive, and 
this is consistent with the objective of reinforcing the crews with positive feedback. In contrast, only a 
very small percentage of the discussion by instructors and crews was directed to problematic aspects 
of crew performance or ways to improve performance, even though instructors tended to hold longer 
sessions for crews whose LOFT performance they rated as lower. This lack of emphasis seems 
inconsistent with the objectives of LOFT. 

The content of the instructors' utterances and the content of the crews' utterances were highly 
correlated along most dimensions examined. Although correlation does not necessarily imply 
causality, our subjective impression is that the general content and emphasis of the debriefings was 
driven almost exclusively by the instructors. This impression is supported by the pattern of discourse, 
discussed below. 

5.1.3 Instructor characteristics. Instructors generally talked substantially more than any of 
the crew members, averaging 61% of the words in debriefings of two-member crews and 49% of the 
words in debriefings of three-member crews. (However, the range of this variable was striking: 
among debriefings of three-member crews, one instructor did 17% of the talking and another 
instructor did 87% of the talking.) The total amount of talking by all crew members combined is, by 
definition, the amount not done by the instructor and thus the two variables are forced into perfect 
negative correlation. However, the fact that the amount of talking done by the instructor is also 
significantly negatively correlated with the amount done by each crew member separately suggests that 
too much talking by the instructor discourages participation by the crew members. Consistent with this 
inference, the amount of talking done by the instructors was significantly negatively correlated with 
other measures of crew participation: words per utterance, number of SI statements, number of 
analyzing utterances, number of proactive questions, depth of crew activity, and extent of analysis and 
evaluation by the crew. (Number of SI statements, number of analyzing utterances, and number of 
proactive questions contribute to the percent crew participation and thus inherently have some degree 
of correlation. These results should be interpreted cautiously.) The average length of utterances by the 
instructors showed a similar pattern of negative correlation with measures of crew participation, 
suggesting that long monologues by the instructor discourage crew participation. 

One might wonder if the percent of participation by the instructor might be driven by the crew; an 
instructor might be forced to do more of the talking if he or she tried unsuccessfully to induce the crew 
to participate substantially. However, the data suggest otherwise: the battery variable Encouragement 
was strongly negatively correlated with percent instructor participation, which is not consistent with 
instructors resorting to lecturing only after seriously attempting to facilitate crew participation. Also, 
our subjective impression is that instructors seemed predisposed to whatever level of facilitation they 
used. 

The large number of questions asked by most instructors suggests that they are attempting to elicit 
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crew participation. The number of questions asked by instructors was not significantly correlated with 
any measures of crew participation, but this might reflect a limitation of the across subjects design of 
this study. An instructor might increase the participation of a given crew by asking more questions, 
but this may be confounded by the possibility that instructors increase the number of questions they 
ask when they encounter a crew that participates inadequately. The crew prone to low participation 
may increase its activity in response to questions but still may remain below average. 

The battery category Questions, which addresses the way in which instructors ask questions and 
takes into account the crew with which the instructor is confronted, appears to be a much more useful 
measure than the simple number of questions the instructors ask. Instructors' scores on the battery 
category were significantly positively correlated with several descriptive measures of crew 
participation and both battery categories of crew participation. 

In all debriefings observed, the discussion revolved primarily around the instructor, even when the 
instructor encouraged the crew to do most of the talking. Direct back and forth discussion among crew 
members was rare; most of the time the pattern was instructor utterance, crew member utterance, 
instructor utterance. 

Many instructors frequently interrupted crew utterances, and in many cases the crew members 
never completed their comment after the interruption. Surprisingly, the frequency of interruption was 
not correlated with any of the descriptive or battery measures of crew participation. Nevertheless, it is 
hard to believe that crew members find frequent interruptions encouraging. 

5.1.4 Crew characteristics. Two important dimensions of crew participation are proactivity 
and analysis of LOFT performance. The descriptive variables do not directly measure these 
dimensions but do shed some light on them. One might expect a proactive participant to ask a lot of 
questions and to initiate topics and issues. However, crew members asked very few proactive 
questions. On the other hand, crew members' words were evenly divided among direct responses to 
the instructor and SI statements (i.e., crew-initiated utterances that add substantively to the 
conversation). Upon further examination, though, it was found that even these SI statements mainly 
address topics initially raised by the instructor. In general, most crew members were willing 
participants who responded readily to the instructor but showed little evidence of proactivity in the 
sense of taking responsibility for the direction of the debriefing. 

On average, individual crew members made only about six utterances per hour that were 
characterized as "analyzing". For coding purposes the definition of "analyzing" was necessarily 
arbitrary, and other definitions might have yielded numbers substantially larger or smaller. 
Nevertheless, this rough characterization suggests substantial room for improvement toward one of the 
major goals of the debriefing. 

Participation by captains and first officers was very similar, as measured by percent participation, 
number of non-directed questions answered, number of proactive questions asked, words per 
utterance, words per response, number of S 1 words, and number of analyzing utterances. (However, 
among two-person crews the percent participation by captains was slightly but significantly greater 
than that by first officers.) On the same variables, flight engineers were generally lower than either 
captains or first officers, although the only difference that reached statistical significance was that 
between first officers and flight engineers on percent participation. 

5.2 Debriefing Assessment Battery 

5.2.1 Battery characteristics. The descriptive variables provide useful information about 
debriefings but are not by themselves adequate to characterize instructor use of facilitation or the nature 
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of crew participation. The Debriefing Assessment Battery was developed to provide a deeper 
characterization of instructor and crew performance. It is designed to be used by raters with a 
substantial background in CRM and a general understanding of the principles of facilitation. High 
interrater reliability was obtained on all categories of this battery with only a moderate amount of 
practice. 

In contrast to reliability, it is difficult to establish the validity of the battery because no standard 
exists with which to compare it. However, the battery does have a certain amount of face validity in 
that the items address behaviors generally agreed upon as necessary for facilitation. Also, the items 
were worded explicitly in terms of the general objectives commonly stated for LOFT debriefings. The 
results discussed below suggest that, in general, the battery does measure what was intended. 

5.2.2 Scores and correlations. Scores on three of the instructor categories — Questions, 
Encouragement, and Focus — were highly predictive of scores on the two categories of crew 
participation. The ability to explore the predictive power of the Introduction category was severely 
limited because of the small variation of instructor scores on this variable; most scores fell on the 
lowest value possible. However, crews scored significantly higher on Analysis & Evaluation in those 
few debriefings in which instructors gave at least a minimal introduction. Also, Introduction scores 
were significantly positively correlated with crew words per utterance, words per response, and 
percent CRM. These data plus the reasons discussed in the beginning of this paper suggest that a 
thorough and explicit introduction is likely to have a substantial effect, although this issue requires 
further study. 

Properly speaking, the use of the video of the crews' LOFT performance is not technically a 
component of facilitation, but it is widely regarded as an important tool that can help the crews 
understand their performance. The nature of the data (transcribed audio tapes of the debriefing) limited 
the types of items that could be used to asses the instructors' Use of Videos. For example, what may 
be one of the most important aspects of the video clips, their content, could not be measured. The 
items in Use of Videos showed little predictive power for any aspect of crew performance except 
percent CRM, and this correlation may only reflect the fact that instructor scores on Use of Videos 
were fairly strongly correlated with instructor percent CRM. Thus we are inclined to delete this 
category from the battery. 

Instructor scores on Questions, Encouragement, and Focus were moderately correlated with 
various descriptive measures of crew participation. Similarly, instructor scores on the battery were 
correlated with some descriptive measures of instructor behavior, and crew scores on the battery were 
correlated with most of the descriptive measures of crew behavior that seemed pertinent. The 
descriptive measures themselves provide at best a partial and largely indirect characterization of 
instructor and crew participation, so the most one could say is that the patterns of correlations are 
consistent with the battery measuring what is intended. For example, as would be expected, crew 
Depth of Activity was somewhat more strongly correlated with percent crew participation than 
Analysis & Evaluation was. Conversely, crew Analysis & Evaluation was more strongly correlated 
with percent crew CRM than Depth of Activity was. 

The battery appears to provide a more meaningful appraisal of instructor facilitation and crew 
participation than most of the descriptive variables do. Also, the descriptive variables require a tedious 
amount of data reduction and can be measured only in a research setting. In contrast, the battery could, 
in principle, be used in real time to evaluate debriefings. We are currently developing a short form of 
the battery that can be used by airline training department personnel to rate instructors and crews 
during observations of their debriefings (McDonnell, Dismukes, & Jobe, in preparation). 

Intercorrelations among Questions, Encouragement, and Focus were high, as was the 
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intercorrelation between crew Analysis & Evaluation and Depth of Activity, thus precluding a 
meaningful factor analysis. Also, the individual items within each category were highly intercorrelated. 
Two possibilities may account for these high intercorrelations: (i) individual items may overlap and/or 
entire categories may overlap substantially in what they measure, and (ii) in this particular data set the 
independent variables measured by the battery items and categories may covary. The latter might 
occur, for example, if instructors tended to either grasp and accept the fundamental concepts 
underlying facilitation or fail to grasp or accept those underlying concepts. Both possibilities may have 
been operating (see discussion of bimodality in section 5.4). In the short form of the battery mentioned 
above, the number of items will be reduced substantially: related items will be combined into one, and 
the content of separate items will be segregated more distinctly. 

5.3 Facilitation Techniques and Common Mistakes 

To facilitate debriefings, instructors used various specific techniques in the broad categories of 
introductions, questions , active listening, and silence. Many instructors showed considerable skill in 
using these techniques; other instructors were markedly less effective, or made little attempt to 
facilitate. Even effective instructors sometimes did things that undercut their efforts at facilitation. 

The most common problem, failing to state explicitly the expectation for crew participation, is 
discussed above. Twenty -eight percent of instructors made no statement at all about expectations and 
only one instructor gave an explicit rationale for why the crew should take an active role. Other 
common mistakes included failing to pause when the crew did not respond immediately to questions, 
keeping the discussion centered on the instructor instead of encouraging the crew to interact with each 
other, making long soliloquies, evaluating crew performance before eliciting crew self-evaluation, 
failing to push beyond superficial description of events, and not getting the crew to analyze why things 
went well. 

A companion to this report describes in detail specific techniques instructors used and suggests 
ways to integrate these techniques for effective facilitation (McDonnell, Jobe, & Dismukes, in press). 
This companion report, written as a training manual for instructors, also suggests ways to avoid 
common facilitation mistakes. 

5.4 Implications for Training 

The fact that instructors' scores on Introduction were uniformly low, much lower than on other 
categories of facilitation, indicates that this is an area in which instructors have not been adequately 
trained. It seems a matter of common sense that if one wants crews to participate in a certain way , 
particularly if that way differs substantially from traditional practice, it is necessary to tell crews 
explicitly what is expected of them. It may be that instructors are so accustomed to the idea that crews 
should be participating proactively that they overlook the fact that this expectation has not been stated 
explicitly to the crews. Alternately, some instructors may have reservations about the concept that it is 
preferable for the debriefing to revolve around the crew, and thus they do not explain this concept to 
the crews. Regardless, a good introduction is easy to provide once instructors recognize its 
importance; thus, training departments may be able to improve crew participation with relatively little 
effort by emphasizing this topic to instructors. Ideally, the introduction should describe how the 
debriefing will be conducted, explain how the crew is expected to participate and what the instructor s 
role is, and provide an explicit rationale for the benefits of crew-centered debriefings. 

The fact that instructor scores on Questions, Encouragement, and Focus were distinctly bimodal 
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and highly intercorrelated suggests that the instructors either grasped the concept of facilitation and 
were able to put it into practice or did not grasp the concept and were therefore unable to practice it 
effectively. Alternately, the instructors who were not effective facilitators may not have "bought into" 
the concept of facilitation or might simply have lacked the ability for this type of approach. 

These findings suggest that the airlines face an issue of standardization and quality control of 
debriefings. Although no attempt was made to measure these characteristics, it was clear that the great 
majority of instructors were highly competent technically, were conscientious, and displayed strong 
interpersonal skills. All seemed comfortable with and committed to the concepts of CRM. Thus, the 
variability may reflect inadequate training of instructors in the techniques of facilitation. When 
interviewed, several instructors spontaneously volunteered that they did not feel adequately trained to 
facilitate. To date, in most airlines with which we are familiar, training in facilitation is vague, 
consisting mainly of general concepts and adages (e.g., "Don't insist on closure"). However, 
facilitation, especially because it departs radically from the instructional techniques traditionally used in 
aviation, requires hands-on training in which instructors observe expert facilitators, practice 
facilitating, and receive feedback. 

As this report is being written, several airlines are expanding their training in facilitation, and this 
can be expected to improve the conduct of debriefings. Currently, an industry group, the ATA AQP 
LOFT/Instructor Focus Group, is preparing a paper that will provide guidance on training instructors 
in facilitation, evaluation of crew performance, and related topics. 

These findings also suggest that the concept of crews debriefing themselves using the instructor as 
a resource (a concept expressed frequently in the CRM literature and in AC 120-35C), though a 
worthwhile goal, is rather idealistic. Only one of the instructors observed attempted to have the crew 
lead their own debriefing. Though that debriefing was one of the better ones in terms of the level of 
crew participation, the crew only partially understood what constituted a good debriefing and needed 
considerable help. In order for crews to take greater responsibility for the debriefing they must first be 
told how to conduct one. It would also help if crews could observe another crew debriefing 
themselves effectively; this could be the subject of classroom training that precedes the LOFT. Crews 
may need to practice self-debriefing of several LOFTs before they become proficient. 

At the current state of industry practice, instructors who attempt to encourage crews to self-debrief, 
or to at least take greater responsibility for the direction of the debriefing, will encounter widely 
varying levels of crew responsiveness. McDonnell et al. (in press), drawing upon a concept expressed 
by Continental Airlines (1992), suggest that facilitation can be conducted at a high, medium, or low 
level, depending on the level of initiative and the self-debriefing skill of the particular crew. In high- 
level facilitation the instructor approaches the ideal of assisting the crew in their own analysis. In low- 
level facilitation the instructor leads the debriefing, directs the crew’s attention to critical issues, and 
may need to lecture to insure points are understood, but the instructor still attempts to foster as much 
self-discovery as possible. 

Instructors are encouraged to attempt to facilitate at the highest level possible for each crew. 
Realistically, however, most crews do not yet have the skills and motivation needed to lead their own 
debriefings without substantial assistance from the instructor. It may be possible to change this 
situation over time if LOFT instructors consistently encourage crews to take a proactive role in 
debriefing their own training and to consider the benefits of debriefing line operations. 

Instructors sometimes mistakenly assume that using facilitation requires giving up their role as 
teacher in the debriefing. On the contrary, good facilitation in no way precludes the instructor from 
adding his or her own perspective to the discussion or from teaching specific points about CRM and 
technical issues as appropriate. Effective facilitators can integrate their teaching points into a group 
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discussion in which the crew members are full participants. 

With the exception of Introduction, instructors' scores on the facilitation categories averaged 
around 4 (adequate), as did crews' scores on Analysis & Evaluation and Depth of Activity. These 
values have little absolute meaning because they depend on the necessarily arbitrary anchoring of the 
scales. Each training department must establish its own standards for satisfactory performance and 
anchor their ratings accordingly. What the Debriefing Assessment Battery provides is a tool for 
evaluating the relative performance of instructors and of crews in LOFT debriefings. 

It has been a matter of faith among training departments that facilitation is an effective tool to 
encourage crews to analyze their performance in LOFT along CRM dimensions in a way that will 
benefit them in line operations. This study provides empirical evidence that this faith is correct. 


6.0 CONCLUSIONS AND RECOMMENDATIONS 


These data provide a portrait of how debriefings were being conducted in major U.S. airlines 
during the period of mid 1994 to mid 1995. This sample seems representative of large U.S. carriers, 
although, as this report was being written many airlines were upgrading their training in facilitation 
and this can be expected to improve the effectiveness of debriefings. The following conclusions and 
recommendations reflect both the objective data and our subjective impressions: 

1 . Most instructors attempted to facilitate crew participation, but their success ranged from very 
good to poor. The bimodal distribution of instructors' battery scores suggests that at least half of the 
instructors grasped and utilized the concept of facilitation effectively, however a substantial minority of 
instructors were consistently ineffective in all measures of facilitation. Almost all instructors appeared 
to be highly competent and conscientious in the traditional role of instructors, thus this variability 
seems to reflect differences in how well instructors comprehend or buy in to the concept of facilitation. 

2. Instructors effectively used a range of specific techniques to facilitate crew participation 
(described in detail in McDonnell et al, in press). Perhaps unwittingly, many instructors also did 
things that appeared to inhibit crew participation. The most striking shortcoming was that most 
instructors made little effort to convey to the crew that they should be proactive, and it is not clear 
whether instructors themselves grasped this concept. It appears that instructors could substantially 
improve crew participation by explicitly explaining the relative roles of the crew and the instructor at 
the beginning of the debriefing. 

3. This study provides empirical evidence that facilitation, when used effectively, substantially 
increases the depth of crew participation and the quality of crew analysis and evaluation of their 
performance. 

4. Crews were generally responsive but showed limited proactivity. Typically, instructors did 
most of the talking and the discussion invariably centered around the instructor’s questions, 
comments, and choice of topics, even when the crew did most of the talking. Most, but not all, 
debriefings emphasized CRM and LOFT performance appropriately. Most debriefings would have 
been improved by greater depth of analysis and more attention to ways to improve performance. 

5. Within each of the five airlines, instructors varied widely in their conduct of debriefings, 
especially in terms of emphasis on CRM, emphasis on crew participation, and effectiveness in 
facilitation. Not surprisingly, the character of crew participation varied similarly, and consequently it 
seems likely that how much the crews learned from the LOFT experience may also have varied 
considerably. This suggests a need for better standardization within companies. The great variability 
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within individual airlines obscured the statistical significance of differences observed among the 
airlines. 

6. These findings suggest that instructors need better training in facilitation. One way to enhance 
training would be to emphasize hands-on practice and to follow up with mentoring by instructors who 
are themselves expert facilitators. The current literature on facilitation is rather idealistic, and 
instructors may become discouraged when they discover that crews sometimes do not immediately 
respond as desired. Instructor training should address obstacles to effective facilitation and should 
provide specific techniques to use when crews do not initially respond. Training should explain to 
instructors that facilitation can be conducted at different levels ranging from predominantly crew-led, 
with instructor assistance, to predominantly instructor-led, but still emphasizing self-discovery by the 
crew as much as possible. Instructors should adapt their level of facilitation in response to the skill and 
responsiveness of the particular crew. 

7. The average session length of about 31 minutes appeared to limit the thoroughness and depth of 
the debriefings. Longer sessions would allow coverage of more issues and greater depth of 
discussion. We have no data on what duration would be optimal, but suggest that an hour might be a 
useful rough target, with adjustments for the needs of individual crews. However, this is a policy 
issue and each airline will have to make its own cost-benefit analysis. 

8. Although we collected no data to assess the effect of the common practice of conducting 
maneuver practice between the LOFT and the debriefing, we suspect that it appreciably impairs the 
ability of the crew to remember and leam from what happened in the LOFT. We recommend that this 
issue should be investigated empirically. 
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H l (80%) □ 3 (4%) □ 5 (1%) 

B2(ll%) H4(2%) 0 6+(l%) 


Figure 1. Crew interaction chart. 


Note: Crew interaction is measured by counting the number of crew utterances between IP utterances. Two or more 
sequential crew utterances indicate interaction occurred, while single crew utterances indicate that there was no interaction. 
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Figure 2. Effect of instructor facilitation on crew analysis and evaluation. 


Note. Instructor Facilitation is a combined measure of Questions, Encouragement, and Focus 
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Frequency 



Instructor Scores 

1 = Poor; 4 = Adequate; 7 = Outstanding 
Figure 3. Distribution of instructor scores on the Debriefing Assessment Battery. 
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Table 1 . Number of Debriefings Observed and Analyzed 



Airline V 

Airline W 

Airline X 

Airline Y 

Airline Z 

Total 

2-person 

6 

0 

5 

5 

9 

25 

3-person 

2 

4 

0 

4 

1 

11 
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Table 2. Interrater Reliabilities for the Debriefing Assessment Battery 


Battery variables 

N 

Pearson’s r 

ip 

Introduction 

35 a 

.91 

Questions 

36 

.78 

Encouragement 

36 

.80 

Focus 

36 

.84 

Use of Videos b 

18 c 

.77 

Crew 

Analysis & Evaluation 

36 

.78 

Depth of Activity 

36 

.73 


a The audio recording began late for one session. 

b Reported reliability for Videos is for crews Y and Z only. Reliability could not be calculated for all crews because one 
item was changed after scoring was completed, and that item was recoded by only one rater. 
c The video equipment was not working for one of the 19 crews in Airlines Y and Z. 
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Table 3. Average Duration of Debriefings (minutes) 
Mean (SD) 


Airline V 

Airline W 

Airline X 

Airline Y 

Airline Z 

Combined 

Airlines 

28.1 (14.8) 

29.2 (2.9) 

40.3 (25.5) 

36.9 (15.6) 

23.1 (7.3) 

30.7 (15.2) 


Note. Differences among airlines were not statistically significant. 
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Table 4. Participation in Debriefings (percent of instructor and crew words) 

Mean (SD) 


Combined 



Airline V 

Airline W 

Airline X 

Airline Y 

Airline Z 

Airlines 

Instructor: 

2- person crews 

3- person crews 

58(15) 

50(3.5) 

58(27) 

61(18) 

54(16) 

40(16) 

67(14) 

41 

6 1 ( 1 5 ) a 
4 9 ( 2 0 ) a 

Captain: 

2- person crews 

3- person crews 

19(6.9) 

23(17) 

16 (8.9) 

24(8.2) 

22(8.1) 

22(7.9) 

19(8.6) 

21 

21(7. 8) b 
20(9.4) 

First Officer: 

2- person crews 

3- person crews 

23(9.4) 

16(12) 

13(9.2) 

15(10) 

23(13) 

27(14) 

14(7.0) 

23 

18(9.7) 
19(1 3) c 

Flight Ener: 
3-person crews 

12(2.8) 

14(11) 


12(7.9) 

15 

13(7.8) 


Note: Differences among airlines were not statistically significant. Significant differences among participants: 
a Instructor > captain, first officer, flight engineer (pc.Ol); ^ captain > first officer (p<.01); c first officer > flight 
engineer (pc. 03). 
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Table 5. Content of Debriefings (percent of instructor and crew words) 

Mean (SD) 



Airline 

V 

Airline 

W 

Airline 

X 

Airline 

Y 

Airline 

Z 

Combined 

Airlines 

Instructor 

CRM 

32(25) 

19(15) 

27(13) 

56(13) 

64(17) 

4 5 ( 2 4 ) a 

Technical 

22(14) 

13(11) 

38(10) 

8. 1(8. 7) 

10(15) 

16(1 5) b 

Mixed 

24(8.6) 

33(13) 

9.8(16) 

5. 6(5. 3) 

6. 2(8. 3) 

1 4 ( 1 4 ) c 

Non-specific 

22(11 ) 

34(12) 

26(7.6) 

30(6.8) 

20(10) 

2 5 ( 1 0) d 

Crew 

CRM 

25(12) 

25(17) 

36(20) 

68(13) 

68(19) 

4 9 ( 2 5 ) e 

Technical 

21(11) 

10(4.2) 

23(8.6) 

5. 6(5. 3) 

6.9(10) 

1 2(1 l) f 

Mixed 

38(13) 

46(12) 

8.8(10) 

11(10) 

14(12) 

21 (1 8)8 

Non-specific 

16(11) 

18(4.6) 

32(14) 

16(7.4) 

12(13) 

17(1 2) h 


Note. Statistically significant differences were found among airlines: a Y>W; Z>V,W,X. b X>Y,Z. c V>Y,Z; 
W>X,Y,Z. d not statistically different. e Y>V,W,X; Z>V,W,X. f V>Y,Z; X>Y,Z. S V>X,Y,Z; W>X,Y,Z. h X>Z. 
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Table 6. Discussion of Crew Performance 
Mean (SD) 


Combined 



Airline V 

Airline W 

Airline X 

Airline Y 

Airline Z 

Airlines 

Positive aspects 
% of IP words 
% of crew words 

19(11) 
6. 5(7. 3) 

5.8(5. 1) 
3. 8(5. 6) 

15(9.3) 

7.4(13) 

16(13) 
9. 9(8. 9) 

24(12) 

9.5(12) 

18(12) 
8. 0(9. 6) 

Negative aspects 
% of IP words 
% of crew words 

3. 8(2.7) 
6.6(4. 1) 

3. 3(2. 5) 
8. 0(7. 9) 

9.4(13) 

9.8(12) 

1. 1(2.1) 
5. 1(3.8) 

1.6(2. 6) 
3.4(7. 2) 

3. 2(5. 5) 
5. 9(6. 7) 

Wavs to improve 
% of IP words 
% of crew words 

5. 0(4. 4) 
3. 6(4.3) 

4. 5(5. 3) 
5. 0(8. 7) 

6. 8(6.7) 
5. 6(4.0) 

3.0(3. 2) 
4.6(5. 1) 

2.7(4. 4) 
5.6(8. 6) 

4. 1(4.6) 
4.8(6. 1) 

Neutral description 
% of IP words 
% of crew words 

18(14) 

40(15) 

17(9.6) 

36(15) 

9. 4(4. 5) 
25(18) 

21(7.0) 

28(15) 

15(8.1) 

33(26) 

17(9.5) 

33(19) 

Performance total 
% of IP words 
% of crew words 

46(21) 

56(22) 

30(14) 

53(19) 

41(15) 

47(17) 

41(13) 

48(21) 

43(13) 

56(27) 

41(15) 

52(21) 


Note. Differences among airlines were not statistically significant. 
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Table 7a. Correlations Between Instructor and Crew Topics 



Instructor variables 

Crew Variables 

% words CRM 

% words technical 

% words CRM 

7 g* * * 


% words technical 

_ * * * 

.85*** 


*p < 05. **p < .01. ***p < .001 . 


Table 7b. Correlations Between Instructor and Crew Emphasis 
on Aspects of Crew Performance 




Instructor variables 


Crew Variables 

positive aspects 

negative aspects 

ways to improve 

positive aspects 

.35* 

-.30 

-.32 

negative aspects 

-.28 

6 1 * * * 

.53** 


ways to improve 


-.04 


.35* 


67 * * * 



Table 8a. Instructor Questions: Two-person Crews 
Mean (SD) 


Airline V Airline W Airline X 

Airline Y 

Airline Z 

Combined 

Airlines 

Number of directed auestions Der hr: 






toCA 18(21) 

21(7.6) 

25(1 

7) 

9.3(12) 

17(15) 

to FO 8. 6(6. 6) 

13(7.6) 

20(1 

0) 

9. 0(7. 3) 

12(8.5) 

Number of non-directed questions Der hr: 






32(19) 

12(17) 

14(3 

• 6) 

19(12) 

20(15) 

Total number of questions Der hr: 

59(27) 

46(26) 

58(27) 

37(14) 

48(23) 


Table 8b. Crew Responses to Non-directed Questions: Two-person Crews 

Mean (SD) 


Airline V Airline W Airline X 

Airline Y 

Airline Z 

Combined 

Airlines 

Percent non-directed questions answered: 

by CA 63(32) 

31(29) 

77(15) 

58(19) 

58(27) 

by FO 53(13) 

35(32) 

60(35) 

51(21) 

50(25) 


Note. Significant differences were found among airlines in percent of non-directed questions answered by CA: Y>X. 
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Table 9a. Instructor Questions: Three-person Crews 
Mean (SD) 


Airline V 

Airline W 

Airline X Airline Y 

Airline Z 

Combined 

Airlines 

Number of directed questions per 

hr: 




toCA 43(31) 

4. 5(6.4) 

7.6(7. 1) 

9.3 

13(20) a 

to FO 20(11) 

4. 7(2. 9) 

6.6(5. 8) 

2.3 

8.5(8. 1) 

toFE 27(2.1) 

5. 6(1.4) 

6. 4(9. 2) 

12 

10(10) b 

Number of non-directed questions 

per hr: 




82(55) 

12(5.2) 

15(9.5) 

16 

27(35) 

Total number of questions per hr: 





171(70) 

27(14) 

35(22) 

39 

59(65) c 

Note. Significant differences were found amon° airlines in a questions directed to CA 
^questions directed to FE: V>WY; c total number of questions per hour: V>WY. 

£ 

A 

> 



Table 9b. Crew Responses to Non-directed Questions: Three-person Crews 

Mean (SD) 



Airline V 

Airline W Airline X 

Airline Y 

Airline Z 

Combined 

Airlines 

Percent non-directed questions answered: 

by CA 51(16) 68(28) 

69(28) 

14 

65(25) 

by FO 

38(28) 

35(47) 

48(36) 

43 

41(36) 

by FE 

26(5.7) 

18(21) 

26(18) 

14 

23(17) 


Note. Percent of non-directed questions answered by FE fell just short of being significantly lower than CA and FO 
answers (p < 0.06; Wilcoxan Matched-pairs test). Other differences among crew members were not significant. 
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Table 10. Percent of Total Crew Words & Utterances Coded R, SI, S & Q 


Crew 


Percent of total words 



Percent of utterances 


R 

SI 

S 

Q 

R 

SI 

S 

Q 

V 

41 

48 

7 

4 

35 

28 

30 

7 

W 

35 

51 

8 

6 

23 

32 

36 

10 

X 

39 

48 

9 

4 

26 

30 

37 

7 

Y 

45 

44 

7 

4 

32 

29 

31 

8 

Z 

54 

38 

5 

3 

40 

32 

22 

6 

All 

44 

45 

7 

4 

33 

30 

30 

7 


^ Response = first responsive utterance by each crew members following a Question. SI = all self-initiated, substantive crew 
statements that raise issues, introduce topics, or add information to an existing topic. Statements = all utterances that do not 
fit the criteria for R, SI, or Q. Question = any utterance that explicitly asks a question. 
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Table 11. Distribution of Crew Questions (number per category) 



CRM 

Technical 

Mixed 

Non-specific 

Total 

Proactive 

7 

1 1 

3 

14 

35 

Reactive 

4 

3 

0 

26 

33 

Miscellaneous 

0 

2 

1 

27 

30 

Total 

1 1 

1 6 

4 

67 

98 
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Table 12. Average Number of Proactive Questions Per Hour 

Mean (SD) 



Airline V 

Airline W 

Airline X 

Airline Y 

Airline Z 

Combined 

Airlines 

CA 

4. 9(3. 6) 

1 .7(2.1 ) 

7. 5(8. 5) 

1.5(1.7) 

1 .2(1.7) 

3.0(4. 3) 

FO 

5. 4(4.0) 

3. 8(3. 2) 

1.1(1. 7) 

2. 5(3. 2) 

2. 1(3. 7) 

3.0(3. 5) 

FE 

8. 1(2.0) 

1.1(1.2) 

-- 

1.3(1.4) 

0 

2.5(3. 2) 


Note. No statistically significant differences were observed between two and three person crews. Statistically significant 


differences found among airlines: CA: X>Z; FE: V>WY. 
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Table 13. Additional Measures of Crew Participation 
Mean (SD) 



Captain 

First Officer 

Flight Engineer 

Crew Average 

Analyzing utterances 
per hour 

7.0 (6.2) 

6.4 (6.1) 

3.4 (2.8) 

6.2 (4.7) 

Words per utterance 

21 (10) 

24 (13) 

17 (9.2) 

22 (10) 

Words per response 

29 (17) 

35 (29) 

21 (9.8) 

30 (17) 


Note. No statistically significant differences were found between airlines or crew positions. 
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Table 14. Debriefing Assessment Battery Scores 
Mean(SD) 


Combined 

Airline V Airline W Airline X Airline Y Airline Z Airlines 


Instructor Profile: 


Introduction 

1 . 5 ( . 6 5 ) 

1 . 4 ( . 7 3 ) 

Questions 

3. 9(1. 7) 

3. 1(1. 9) 

Encouragement 

3.8(1 .7) 

3. 5(2. 4) 

Focus 

3.2(1 .8) 

2.9(1 .0) 

Use of Videos 

-- 

4 . 3 ( . 8 5 ) 

Crew Profile: 

Analysis & Eval. 

3.3(1 .3) 

3.4(1 .2) 

Depth of Activity 

4. 0(1.0) 

4. 2(1. 5) 


1-1(13) 

2. 1(1. 3) 

1.4(.42) 

1 . 6 ( . 8 3 ) 

3. 4(1. 5) 

5.0(.66) 

4. 2(2.0) 

4. 1(1. 6) 

3.3(1 .7) 

5.1 (.66) 

3. 9(2.0) 

4. 1(1. 7) 

3.0(1 .3) 

5.0(.69) 

4.0(1 .7) 

3.8(1 .6) 

2.9(.62) 

4. 5(1. 4) 

5.1(1 .0) 

4.4(1 .2) 


3.3(1 

1) 

4.8(.87) 4. 2(1. 8) 

3.9(1 

-4) 

4.0(1 

.5) 

5. 1(1.1) 4. 4(1. 9) 

4.4(1 

-4) 


Note: Numbers are average scores of two independent raters (except Video scores for airlines W & X, which were coded 
by only one rater) on a 7-point Likert scale: 1 = poor, 2 = marginal, 3 = needs improvement, 4 = adequate, 5 = good, 

6 = very good, 7 = outstanding. 

No differences between airline average scores were statistically significant. 
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Table 15. Frequencies of Rating Scores on the Debriefing Assessment Battery 






Rating Scores (Average of the two raters) 


Subjective 

variables 

N 

Poor 

Margi 

Needs 

inal Improve 

Adequate 

Good 

Very 

Good 

Outstanding 

ip 

Introduction 

35 

23 

8 

3 

0 

1 

0 

0 

Questions 

36 

2 

7 

4 

3 

9 

11 

0 

Encouragement 

36 

2 

9 

2 

4 

9 

9 

1 

Focus 

36 

2 

7 

4 

6 

10 

7 

0 

Use of Videos 

26 

0 

3 

4 

6 

5 

6 

2 

Crew 

Analysis & Eval. 

36 

1 

6 

8 

4 

13 

3 

1 

Depth of Activity 

36 

1 

2 

8 

5 

11 

7 

2 
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Table 16. Spearman Correlations Between IP and Crew Variables on the 
Debriefing Assessment Battery 


Instructor variables 3 


Crew variables 3 

Introduction 

Questions 

Encourage 

Focus 

Videos 

Analysis & Evaluation 

.28 

75 *** 

7g *** 

75 *** 

.33 

Depth of Activity 

.13 

59 *** 

78 *** 

5 ] *** 

.26 


a See Debriefing Assessment Battery (Appendix C) 
*p < -05. **p < .01. ***p < .001 . 
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Table 17. Spearman Intercorrelations Among Instructor Variables: 
Debriefing Assessment Battery 


Subscales 

Questions 

Encouragement 

Focus 

Use of Videos 

Introduction 

.55*** 

, 44 * * 

49 ** 

.29 

Questions 

— 

90*** 

g9* * * 

.51** 

Encouragement 


- 

7g* * * 

.45* 

Focus 



— 

VO 

C*~i 

Use of Videos 




- 

*P < -05. **p < .01. 

***p < . 001 . 
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Table 18. Relationship of High and Low Introduction Scores to 
Crew Analysis & Evaluation and Depth of Activity 


Mean (SD) 

Introduction Scores 

N 

Analysis & Evaluation 

Depth of Activity 

1.0 

10 

3.2 (1.3)* 

4.1 (1.4) 

1.8 - 4.9 

9 

4.4 (.63)* 

4.6 (1.0) 


Note. The ten debriefings for which instructor Introduction scores were lowest were compared with the nine debriefings 
for which Introduction scores were highest. 

*p < .025, t-test 
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Table 19. Correlations Between Instructor Battery 3 and Descriptive 15 Variables 


Descriptive variables 


Battery 

Variables 

% total 
participation 

Words per 
utterance 

# directed 
questions 

# non- 
directed 
questions 

Total # 
questions 

% words 
addressing 
performance 

% words 
addressing 
CRM 

Introduction 

-.07 

.12 

. 41 * 

-.20 

. 42 * 

.05 

. 35 * 

Questions 

_ 49 ** 

-.38* 

. 56 *** 

.10 

. 60 *** 

.05 

. 35 * 

Encourage 

- 75 ** * 

-.58*** 

. 38 * 

.15 

. 43 ** 

-.04 

.25 

Focus 

-. 40 * 

-.31 

. 50 ** 

.08 

52 *** 

.12 

. 45 ** 

Use of Videos 

-.06 

.09 

.24 

.17 

.38 

.25 

69 *** 


a See Debriefing Assessment Battery (Appendix C) 
b See Appendix E 

*p< .05. **p< .01. ***p< .001. 
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Table 20. Correlations Between Instructor Battery Variables and Crew Descriptive Variables 


Crew Descriptive variables 


Instructor 

Battery 

variables 

Percent 

participation 

Words per 
utterance 

Words per 
response 

Self- initiated 
words 

Analyzing 

utterances 

Proactive 

questions 

Percent 

CRM 

Introduction 

.07 

o * * * 

.35 * 

-.06 

.12 

-.08 

.45** 

Questions 

.49** 

.42* 

.28 

.18 

.56*** 

-.07 

.56*** 

Encourage 

74 * * * 

.50** 

.34* 

47 ** 

.70*** 

.10 

* 

o 

Focus 

4^ 

O 

* 

.39* 

.28 

.09 

.53** 

-.16 

63 *** 

Videos 

.05 

.31 

.11 

-.02 

.14 

-.21 

67 * * * 


*p < .05. **p < .01. ***p<.001. 
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Table 21. Correlations Between Crew Battery and Descriptive Variables 


Descriptive variables 


Battery 

variables 

Percent 

participation 

Words per 
utterance 

# of words 
per response 

Self-initiated 

words 

Analyzing 

utterances 

Proactive 

questions 

Percent 

CRM 

Analysis & 
Evaluation 

67 * * * 

.58*** 

.50** 

5 ] * * * 

80*** 

-.14 

.56** 

Depth of 

# 84 * * * 

5 7 * * * 

.45** 

76** * 

go*** 

.10 

.34* 


Activity 


*p < .05. **p < .01. ***p < .001. 
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Table 22. Correlations Between Instructor Descriptive Variables and 
Crew Battery and Descriptive Variables 


Instructor variables 

Crew variables 

% 

participation 

Words per 
utterance 

# of directed 
questions/hr 

# of non- 
directed 
questions/hr 

% words 
addressing 
performance 

% words 
addressing 
CRM 

% participation 

-,99 a 

_ g9*** 

.08 

.23 

-.06 

-.05 

Words per utterance 

-.38* 

.07 

.07 

-.16 

.17 

.39* 

Words per response 

-.19 

.20 

-.06 

-.24 

.14 

.36* 

SI statements 
(# words per hour) 

_ 79 *** 

- (59 * ** 

-.07 

.20 

-.07 

-.06 

# of analyzing 
utterances per hour 

-.65*** 

-.35* 

.19 

.09 

.08 

.23 

# of proactive 
questions per hour 

-.31* 

_ 47 * * 

.07 

.24 

-.41* 

-.27 

% words addressing 
CRM 

-.04 

.17 

.17 

-.28 

.24 

.76*** 

% words addressing 
performance 

.08 

.18 

.37* 

-.10 

.41* 

.08 

Analysis & Evaluation 

- 67*** 

-.39* 

.23 

.05 

.12 

.40* 

Depth of Activity 

- g4*** 

-.55** 

-.001 

.09 

.02 

.28 


a Forced correlation; see discussion. 


*p < .05. **p < .01. ***p < .001. 
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Table 23, Variability Within and Across Instructors 
Mean (SD) 


Variables 

Average value 
of variable 

Delta scores 

t- value 

p- value 



same instructor 
(n=4) 

different instructor 
(448 random pairings) 



Duration 

30.7 (15.2) 

18.2 (1.3) 

13.7 (12) 

0.67 

n.s. 

IP % CRM 

45 (24) 

22.8 (7.0) 

26.9 (18) 

-1.12 

n.s. 

IP % performance 

41 (15) 

21.8 (9.0) 

18.1 (12) 

0.84 

n.s. 

IPQEF 

4.0 (1.6) 

0.73 (0.48) 

1.75 (1.3) 

-4.14 

< .005 
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Appendix A. Coding 


Utterance factors coded 

Utterance length: 

number of words 

Speaker: 

Instructor (IP), 2nd Instructor in role of Flight Engineer (FEI). 
Captain (CA), First Officer (FO), or Flight Engineer (FE) 

Interruptions/Inteij ections : 

Completed (C), Unfinished (U), Interrupted (I), Interrupted 
and Unfinished (I/U), Active listening interjection (I/AL) 

Utterance type: 

Question, Command, Response, or Statement (Statements self- 
initiated by crew further coded as SI) 

Target of Question (if clearly 
directed to a particular crew 
member): 

Captain (CA), First Officer (FO), or Flight Engineer (FE) 

Crew Proactive Questions: 

"P" if crew question is proactive, "O" (Other) if it is a reactive 
or miscellaneous question 

Topic type: 

CRM, Technical, Mixed (CRM & Technical), or Non-Specific 

Analysis: 

"A" if crew analyzes situation/performance, "O" (Other) if not 

Evaluation of crew 
performance: 

Positive. Negative. Improve, or Neutral 

Video factors coded 

ON ( ): 

All video segments are coded by indicating segment number 
with duration in parentheses [e.g., ON #1 (:45)] 

OFF: 

Code end of video segments by indicating (OFF) 

SEARCH ( ): 

Time spent searching in silence [e.g., SEARCH (:30)] 
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CODING RULES 


Utterance Length (LENGTH) 

1 . Fill in a word count for every utterance for which a speaker and content are identified. Do not count 
utterances in which speaker is identified but the words are unintelligible; or words are transcribed 
but speaker cannot be identified. 

2. Count repeated words (i.e. , stuttering) as one word only. 


Speaker (SPKR) 

Identify the speaker of each utterance using one of the following; IP, CA, FO, FE, or FEI. 


Transcribing Utterances (UTTERANCE) 

1 . Transcribe the audiotape verbatim. 

2. Record all pauses 3 seconds or longer in bold type. 

3. Type titles in parentheses [e.g., (CA) or (FO)] in place of spoken names and type (XX) in place of 

spoken name of airline. 

4. If an utterance is phrased as a statement but is intended to evoke a response, end the utterance with 

a "(?)" so it can be coded as a command. 

5. If a speaker is interrupted (inteijections of active listening or brief interruptions which do not 
change the flow of the original speaker's utterance) or is talked over but clearly continues on to 
complete the sentence or thought, transcribe and code the continuation(s) as part of the initial 
utterance with "(x)" where the interruption or interjection occurs, and type and code each 
interrupting utterance separately below ("1” in the INT column). 

6. If speaker is interrupted by a substantial utterance and continues, but the topic or flow is slightly 

altered, code the initial utterance as unfinished ("U" in the INT column), and transcribe and code 
the continuation as a separate utterance after the interrupting utterance. 

7. If a speaker makes a statement and then asks a question during a single speaker turn, break it into 
two separate utterances where the question begins. 

8. If a speaker clearly changes topics in the middle of a single speaker turn, transcribe and code the 
topic change as a separate utterance. 

9. Record length of video silent search time (no one speaks while DP tries to find a specific video 
segment) in bold type. 


Interruptions / Inteijections (INT) 

1. Code all utterances that are not completed (whether the speaker is interrupted or trails off) as "U" 
and code all completed utterances as "C" 
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2. Code all utterances that interrupt or interject the preceding speaker as "I" (code as "I/U" if the 
interruption is not completed, either because the preceding speaker keeps talking or another 
speaker interrupts the interruption) 

3. Code all active listening as "AL" (code interjections of active listening as "I/AL") 

Utterance Type (TYPE! 

Question = Any utterance that explicitly asks a question. 

Command = Any IP utterance that commands a response but is not phrased in question form. 

Response = First utterance by any or all crew members following a Question or Command, unless 
content of utterance makes it obvious that it is non-responsive. 

SI (crew) = All self-initiated, substantive crew statements that raise issues, introduce topics, or add 
information to an existing topic. 

Statement = All utterances that do not fit the criteria for Q, C, R, or SI, unless content makes it 
obvious that the utterance is responsive (R) to the preceding Q or C (e.g., when separated by an 
intervening utterance). 


Question Target (O TRGT) 

1. Code target of IP question if clearly directed to a particular crew member (e.g., "CA"). 

2. For non-directed IP questions, code the crew member(s) who respond in parentheses [e.g., "(CA)" 
or "(FO,CA)"] or code as "( )" if no one responds 

Crew Proactive Questions IPAOl 

1. Record a "P" in the crew PAQ column if crew question is proactive, or an "O" (other) if the 
question is not proactive (i.e., reactive or misc.) 

Proactive questions include clarification/verification questions used to raise new issues or bring new 
information into the conversation (e.g., "You wanted help?") and questions designed to gather 
information (e.g., "Did we have runway three?") 


Topic Type (TYPE) 

CRM = Pertains to the coordination and interaction of the crew and specifically relates to one or more 
CRM issues or topics. 

Technical = Pertains to specific techniques of flying and navigating the airplane and/or managing the 
systems, without reference to coordination, planning, communication, judgment, or decision 
making among crew members. 

Mixed = Has between 1/3 and 2/3 of both CRM and technical. 

Non-Specific = Does not refer specifically to either CRM or technical topics. Includes undetermined, 
extraneous, procedural, and maintenance of discourse. 
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(ANALYSIS) 


Code all utterances that indicate the speakers are Analyzing the situation &/or their performance in the 
LOFT by considering any of the following issues (both explicit and implicit) as A (Analyzes). 
Code all utterances which are not analytical as O (Other). 

Generally, analyzing utterances are those that go beyond just describing what happened to discussing 
why it happened and identifying what factors contributed to the situation and/or how these factors 
influenced the outcome. 

• explanations of why something was done and/or done a certain way, or what could have been done 

differently. Key words include because, should have, could have , and might have (e.g. . "I think 
we could have performed faster in holding because we had to take a couple of turns in holding just 
to make sure we got set up." and "I felt a little disorganized pushing off and taxiing out and doing 
all of that and then having to de-ice; that breaks your flow because you don’t put the flaps down”) 

• how & why factors influenced decisions, actions, and outcomes (e.g. . "The reason this influenced 

my decision/actions was ..." and "I was thinking this, so I did this"). 

• contingencies ( e.g. . "It might have been a lot different if we had asked for more time before we took 

that turn. Maybe I should have asked for one more minute.") 


(EVALUATION) 

Code all utterances which indicate Evaluation of Crew Performance as follows: 
Pos = positive evaluation of crew performance 
Neg = negative evaluation of crew performance 
Improve = suggestions for ways to improve 
Neut = neutral evaluation of crew performance 
Code all utterances which do not fit into the above categories as O (other) 


(VIDEO) 

Code all video segments by indicating segment number with duration in parentheses [e.g., ON #1 
(:45)], when segment ends (OFF), and time spent searching in silence [e.g., SEARCH (:30)] 


(COMMENTS) 

1. Indicate any pauses IP uses to allow crew to formulate responses to questions, or pauses after crew 
statements which encourage crew to say more. 

2. Indicate use of probing questions to encourage crew to analyze in more depth. 

3. Indicate when IP follows up on topics initiated by crew. 

4. Note any noticeably good or poor IP techniques. 

5. Record any revelations and/or any specific references to video. Also indicate any difficulty using 
video equipment. 
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Appendix B. 



Calculation of utterance variables 

# of words for IP, CA, FO, 
FE, Crew, total : 

number of words spoken by each; add CA, FO, and FE totals 
together for crew total 

% participation: 

# of words per speaker total # of words for the debriefing 

# of analyzing utterances per 
hour for CA, FO, FE, Crew: 

(# of analyzing utterances ■+• duration) x 60 

# of questions per hour for CA, 
FO, FE, Crew: 

(# of questions -*• duration) x 60 

# of proactive questions per 
hour for CA, FO, FE, Crew: 

(# of proactive questions -*■ duration) x 60 

# S 1 words per hour for CA, 
FO, FE, Crew: 

(# of S 1 words + duration) x 60 

# of words per response for 
CA, FO, FE, Crew: 

# of response words + # of responses 

% crew words positive: 

# of crew words positive -s- total # of crew words 

% crew words negative + 
improve: 

# of crew words negative and improve total # of crew words 

% crew words improve: 

# of crew words improve -5- total # of crew words 

% crew words negative: 

# of crew words negative total # of crew words 

% crew words positive + 
negative + improve: 

# of crew words positive, negative, and improve total # of 
crew words 

% crew words neutral: 

# of crew words neutral +■ total # of crew words 

% crew words performance: 

# of crew words performance (positive, negative, improve, and 
neutral) + total # of crew words 

% IP words CRM: 

# of IP words CRM + total # of IP words 

% IP words technical: 

# of IP words technical -s- total # of IP words 

% IP words mixed: 

# of IP words mixed -s- total # of IP words 

% IP words non-specific: 

# of IP words non-specific -s- total # of IP words 

% IP words CRM + half of 
mixed: 

# of IP words CRM + half of mixed + total # of IP words 
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% IP words technical + half of # of IP words technical + half of mixed total # of IP words 
mixed: 

% IP words positive: # of IP words positive + total # of IP words 

% IP words negative + # of IP negative and improve total # of IP words 

improve: 

% IP words improve: # of IP words improve + total # of IP words 

% IP words negative: # of IP words negative +■ total # of IP words 

% IP words positive + negative # of IP words positive, negative, and improve total # of IP 
+ improve: words 

% IP words neutral: # of IP words neutral -4- total # of IP words 

% crew words CRM: # of crew words CRM + total # of crew words 

% crew words technical: # of crew words technical -s- total # of crew words 

% crew words mixed: # of crew words mixed - 5 - total # of crew words 

% crew words non-specific: # of crew words non-specific total # of crew words 

% of crew words CRM + half # of crew words CRM + half of mixed - 5 - total # of crew words 
of mixed: 

% of crew words technical + # of crew words technical + half of mixed total # of crew 

half of mixed: words 

# of questions directed to CA, (# of questions directed to each - 3 - duration) x 60 
FO, FE per hour: 

% of non-directed questions # of non-directed questions answered by each - 5 - total # of non- 

answered by CA, FO, FE, no directed questions 

one 

# of directed questions per (# of directed questions •+■ duration) x 60 
hour: 

# of non-directed questions per (# of non-directed questions duration) x 60 
hour: 

total # of questions per hour (total # of directed questions + total # of non-directed questions 

■ 3 - duration) x 60 

number of video segments (# of segments shown -3- duration) x 60 
shown per hour: 
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average duration of video 
segments shown: 

total duration of all segments shown -^ # of segments shown 

# of times IP interrupts crew 
per hour: 

(total # of IP interruptions - 5 - duration) x 60 

% of crew utterances 
interrupted: 

total # of crew utterances interrupted by IP total # of crew Q, 
R, and S 1 utterances 

% of crew utterances 
interrupted and unfinished: 

# of crew utterances interrupted and unfinished total # of 
crew Q, R, and S 1 utterances 

% of crew utterances 
interrupted and completed: 

# of crew utterances interrupted and completed -s- total # of crew 
Q, R, and SI utterances 

# of crew (question, response, 
and SI) utterances per hour: 

[# of crew (Q, R, and SI) utterances + duration] x 60 

# of words per utterance for IP, 
CA, FO, FE, crew: 

total # of words for each total # of utterances for each 
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Appendix C 


DEBRIEFING ASSESSMENT BATTERY 


INSTRUCTOR PROFILE 

The Instructor Profile is a summary of the strategies and techniques IP's use to assist crews in conducting their own 
debriefings while giving direction and focus as necessary. The two main goals of the debriefing are to 1) get the crew to 
perform an in-depth analysis of the situation that confronted them, how they understood and managed the situation, the 
outcome, and ways to improve, and 2) get the crew to participate in a proactive, rather than reactive, manner in which 
they initiate discussion and elaborate beyond the minimal. These goals are based on the assumption that active 
participation by the crew will result in a higher level of learning and increased likelihood of transfer to the line. 

Directions: 

Use the scale below to rate the instructors on each of the following elements, then total the scores to get the overall 
rating for each category 

Poor Marginal Needs Improvement Adequate Good Very Good Outstanding 

1 2 3 4 5 6 7 


Introduction 

One purpose of the introduction is to let the crew know that participation and self-evaluation are expected of them, and 
why it is important. 

Makes clear that his role is guide/facilitator and that crew should do most of the talking 

Clearly conveys that crew should take an active role, initiating discussion rather than just responding to him 

Clearly conveys that he wants crew to dig deep, critically analyzing the LOFT and their performance 

Gives a persuasive rationale for the crew to participate actively and make their own analysis 

Overall rating of Introduction 


Questions 

The purpose of asking questions is to get the crew to participate, focus the discussion on important topics, and enlist the 
crew in discussing the topics in depth. 

Asks an appropriate number of questions to get crew talking & lead them to issues 

Avoids answering for the crew when they do not respond immediately or correctly and uses a pattern of 

questioning that keeps the focus on the crew 

Uses probing and follow-up questions to get crew to analyze in depth and to go beyond yes/no and brief factual 

answers 

Uses questioning techniques to encourage interaction and sharing of perspectives among crew members 

Overall rating of Questions 
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Encouragement 

Encouragement refers to the degree to which the instructor encourages and enables the crew to actively and deeply 
participate in the debriefing. 

Conveys sense of interest in crew views and works to get them to do most of the talking 

Encourages continued discussion through active listening, strategic pauses, avoiding disruptive interruptions, 

and/or following up on crew-initiated topics 

Encourages all members to participate fully, drawing out quiet members if necessary 

Refrains from giving long soliloquies or giving his own analysis before crew has fully analyzed 

Overall rating of Encouragement 


Focus on Crew Analysis and Evaluation 

The goal of the debriefing session is to get the crew to evaluate and analyze their own CRM performance so they will 
learn more deeply and can gain practice in debriefing themselves, a skill they can then begin to use on the line. 

Encourages crew to analyze along CRM dimensions the situation that confronted them, what they did to manage 

the situation, and why they did it 

Encourages crew to evaluate their performance and/or ways they might improve 

Encourages crew to explore CRM issues and how they specifically affect LOFT performance and line operations 

Encourages crew to analyze issues, factors, and outcomes in depth, going beyond simply describing what 

happened and what they did 

Overall rating of Focus on Crew Analysis & Evaluation 


Use of Videos 

One stated purpose of showing videotaped segments of the LOFT is to enable the crew members to see how they 
performed from an objective viewpoint so they can better evaluate their performance* More realistically, perhaps, the 
video reminds the crew of the situation, aiding their memory and providing a focus for discussion. 

Shows an appropriate number of videos of appropriate duration to illustrate/introduce topics 

Uses video equipment efficiently: is able to find desired segment without wasting time and pauses the video if 

substantial talk begins while playing 

Consistently discusses video segments, using them as a springboard for discussion of specific topics 

Has a point to make and uses the video to make that point. 

Overall rating of Use of Videos 
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CREW PROFILE 

The crew profile measures the degree and depth of participation by the crew. 


Directions: 

Use the scale below to rate the crew on each of the following elements, then total the scores to get the overall rating for 
each category 

Poor Marginal Needs Improvement Adequate Good Very Good Outstanding 

1 2 3 4 5 6 7 


Crew Analysis and Evaluation 

Crew analysis and evaluation refers to the depth to which the crew members analyze the LOFT situation and evaluate 
their performance. 

Analyze along CRM dimensions the situation that confronted them, what they did to manage the situation, and 

why they did it 

Evaluate their performance and ways they might improve 

Explore CRM issues and how they affect LOFT performance and line operations 

Analyze issues, factors, and outcomes in depth, going beyond simply describing what happened and what they 

did 

Overall rating of Crew Analysis & Evaluation 


Depth of Crew Activity 

Activity refers to how actively, versus passively, and deeply the crew participates in and initiates discussion. 

Go beyond minimal responses to IP questions 

Participate deeply and thoughtfully 

Initiate dialogue rather than just responding to questions, and/or interact with each other rather than only with 

the IP 

Behave in a predominantly proactive rather than reactive manner, being actively involved rather than just 

passing through the training 

Overall rating of Depth of Crew Activity 
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Appendix D. 


ANCHORING OF THE DEBRIEFING ASSESSMENT BATTERY 


IP Introduction 


Outstanding: 

- Very specifically and thoroughly explains that his role is guide/facilitator and that crew should do most of the talking and 

lead the discussion 

- Sets strong expectations for proactive crew participation, explicitly stating they should initiate discussion rather than just 
responding to IP questions 

- Explicitly and emphatically states that crew should dig deep, critically analyzing the LOFT and their performance 

- Gives a persuasive rationale for the crew to participate actively and make their own analysis and makes a strong case for 

why it is important to do it this way. 

Very Good: 

- Clearly conveys that his role is guide/facilitator and that crew should do most of the talking and lead the discussion 

- Clearly conveys that crew should take an active role, initiating discussion rather than just responding to IP 

- Clearly conveys that crew should dig deep, critically analyzing the LOFT and their performance 

- Clearly conveys the general rationale for the crew to participate actively and make their own analysis 

Good: 

- Conveys that his role is guide/facilitator and that crew should do most of the talking, but not specifically that they should 

lead their own discussion. 

- Conveys that crew should take an active role, initiating discussion rather than just responding to IP 

- Conveys that crew should dig deep, critically analyzing the LOFT and their performance 

- Makes a general statement of the rationale for the crew to participate actively and make their own analysis 

Adequate: 

- Conveys that his role is guide/facilitator and that crew should do most of the talking, but does not emphasize strongly 

- Conveys that crew should take an active role and initiate discussion 

- Conveys that crew should analyze the LOFT and their performance 

- Gives a clear, though implicit rationale for the crew to participate actively and make their own analysis 

Needs Improvement: 

- Implies that his role is guide/facilitator and that crew should do most of the talking, but does not emphasize strongly 

- Implies that crew should take an active role and initiate discussion 

- Implies that crew should analyze the LOFT and their performance 

- Gives a vague, implicit rationale for the crew- to participate actively and make their own analysis 
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Marginal: 

- Implies that his role is guide/facilitator and that the crew should talk, but does not emphasize 

- Implies that crew should take an active role, but does not specify what they should do. 

- Implies that crew should discuss the LOFT and their performance 

- Gives vague impression of why crew should participate actively 

Poor: 

- Does not make clear that his role is guide/facilitator or that crew should do most of the talking 

- Does not make clear that crew should take an active role or initiating discussion 

- Does not make clear that crew should dig deep or critically analyze the LOFT and their performance 

- Does not give rationale for the crew to participate actively and make their own analysis 
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IP Questions 


Outstanding: 

- Consistently asks questions as appropriate to get crew talking & lead them to issues 

- Consistently rewords questions or otherwise avoids answering for the crew when they do not respond immediately or 
correctly, and consistently uses a pattern of questioning that keeps the focus on the crew 

- Consistently uses probing and follow-up questions as a tool to evoke in-depth discussion and optimize crew' self- 
discovery, while forcing crew to go beyond yes/no and brief factual answers 

- Consistently uses questioning techniques to encourage substantial interaction and sharing of perspectives among crew 

members 


Very Good: 

- Frequently asks questions when appropriate to get crew talking & lead them to issues 

- Predominantly rewords questions or otherwise avoids answering for the crew when they do not respond immediately or 

correctly and predominantly uses a pattern of questioning that keeps the focus on the crew 

- Frequently uses probing and follow-up questions as a tool to evoke in-depth discussion and optimize crew self-discovery, 

pushing crew to go beyond yes/no and brief factual answers 

- Frequently uses questioning techniques to encourage interaction and sharing of perspectives among crew members 


Good: 

- Generally asks questions as necessary to get crew talking & lead them to issues 

- Generally rewords questions or otherwise avoids answering for the crew when they do not respond immediately or 
correctly and generally uses a pattern of questioning that keeps the focus on the crew 

- Generally uses probing and follow-up questions to get crew to analyze in depth and to go beyond yes/no and brief factual 
answers but may steer crew to predetermined answers while emphasizing self-discovery. 

- Generally uses questioning techniques to encourage interaction and sharing of perspectives among crew' members 


Adequate: 

- About half of the time asks questions when necessary to get crew talking & lead them to issues 

- Generally avoids answering for the crew when they do not respond immediately or correctly, but may not reword the 
questions. On average uses a pattern of questioning that keeps the focus on the crew 

- On average uses probing and follow-up questions to get crew to analyze in depth and to go beyond yes/no and brief factual 

answers but steers crew to predetermined answers as much as emphasizes self-discovery. 

- On average uses questioning techniques to encourage interaction among crew members 


Needs Improvement: 

- Sometimes asks questions w'hen necessary to get crew talking & lead them to issues 

- To some extent avoids answering for the crew when they do not respond immediately or correctly and uses a pattern of 

questioning that keeps the focus on the crew 

- Sometimes uses probing and follow-up questions to get crew to analyze in depth and to go beyond yes/no and brief factual 

answers but steers crew to predetermined answers more than emphasizes self-discovery. 

- Sometimes uses questioning techniques to encourage interaction among crew members 
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Marginal: 

- Occasionally asks questions to get crew talking & lead them to issues 

- Occasionally avoids answering for the crew when they do not respond immediately or correctly but generally answers for 

them rather than keeping focus on the crew. 

- Occasionally uses probing and follow-up questions to get crew to analyze in depth but generally settles for yes/no and 

brief factual answers 

- Occasionally uses questioning techniques to encourage interaction among crew members 

Poor: 

- Rarely asks questions to get crew talking or lead them to issues 

- Usually answers for the crew' when they do not respond immediately or correctly. 

- Rarely uses probing and follow-up questions to get crew to analyze in depth. Usually settles for yes/no and brief factual 
answers 

- Rarely uses questioning techniques to encourage interaction among crew members 
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IP Encouragement 


Outstanding: 

- Consistently communicates an interest in crew views and actively strives to get them to do most of the talking and lead 

their own discussion. 

- Consistently uses active listening and pauses, avoids interrupting, and follows up on crew topics. 

- Consistently encourages all members to participate and draws out quiet members as necessary. 

- Consistently refrains from lecturing and giving own analysis before crew. 

Very Good: 

- Clearly communicates to the crew that their views are important and works to get them to do most of the talking and to 

lead their own discussion. 

- Frequently uses techniques such as active listening and pauses, avoids interrupting, and follows up on crew topics to 
encourage continued discussion. 

- Frequently encourages all members to participate and attempts to draw' out quiet members as necessary. 

- Usually refrains from lecturing and giving own analysis before crew. 

Good: 

- Shows a clear interest in crew' views and attempts to get them to do most of the talking. Makes an effort to get crew to 
lead their own discussion. 

- Often uses active listening and pauses, avoids interrupting, and follows up on crew topics. 

- Generally encourages all members to participate, drawing out quiet members as necessary. 

- Sometimes lectures, but generally gets crew to analyze situation before giving own analysis. 

Adequate: 

- On average demonstrates a desire to have crew participate and discuss their views. 

- Uses some facilitation techniques to encourage crew discussion and generally avoids interrupting them. Acknowledges 
crew topics but may not follow up on them thoroughly. 

- Attempts to get all crew members involved. 

- On average gets the crew to analyze the situation themselves before evaluating and lecturing to them. 

Needs Improvement: 

- Shows interest in crew views but does not push them to do most of the talking. 

- Sometimes uses active listening and pauses, and follows up on crew topics, but also sometimes interrupts. 

- Expresses a desire for crew to participate but does not put a lot of effort into getting all members actively involved. 

- Sometimes lectures rather than letting crew do the talking. 
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Marginal: 

- Exhibits only modest interest in crew views. 

- Only occasionally uses active listening, pauses, and/or follows up on crew topics, and often interrupts. 

- Expresses a desire for crew to participate but puts minimal effort into actively encouraging them to do so. 

- Tends to lecture and analyze for crew- without encouraging them to discuss what happened themselves. 

Poor: 

- Gives the impression that crew views are not valued. 

- Frequently hinders rather than encourages crew talk and does not follow up on topics initiated by crew. 

- Makes little attempt to get crew members to participate. 

- Frequently lectures to crew about what they did and how to improve. 
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IP Focus on Crew Analysis and Evaluation 


Outstanding: 

- Continually encourages and pushes crew to analyze along CRM dimensions the situation that confronted them, what they 
did to manage the situation, and why they did it. 

- Consistently encourages and pushes crew to evaluate their performance and/or ways they might improve. 

- Consistently encourages crew to explore CRM issues and how they specifically affect LOFT performance and line 

operations. 

- Continually encourages crew to analyze issues, factors, and outcomes in depth, going beyond simply describing what 

happened and what they did. 


Very Good: 

- Frequently encourages and pushes crew to analyze along CRM dimensions the situation that confronted them, what they 

did to manage the situation, and why they did it. 

- Frequently encourages crew to evaluate their performance and/or ways they might improve. 

- Frequently encourages crew to explore CRM issues and how they specifically affect LOFT performance and line 

operations. 

- Frequently encourages crew to analyze issues, factors, and outcomes in depth, going beyond simply describing what 

happened and what they did 


Good: 

- Generally encourages crew' to analyze along CRM dimensions the situation that confronted them, what they did to manage 

the situation, and why they did what they did, but may settle for less than extensive discussion. 

- Generally encourages crew to evaluate their performance and/or ways they might improve. 

- Generally encourages crew to explore CRM issues, and attempts to get crew to discuss how they specifically affect LOFT 

performance and line operations. 

- Generally encourages crew to analyze issues, factors, and outcomes in depth. Generally encourages crew to go beyond 

simply describing what happened and what they did. 


Adequate: 

- On average encourages crew to analyze along CRM dimensions the situation that confronted them and what they did to 
manage the situation. Encourages but does not push crew to analyze why they did what they did. 

- Tends to encourage crew to evaluate their performance and/or ways they might improve, but may not pursue thoroughly. 

- On average encourages crew to explore CRM issues but tends not to get crew to discuss how they specifically affect both 

LOFT performance and line operations. 

- Generally encourages crew to analyze issues, factors, and outcomes, but settles for moderate depth, sometimes letting 

crew simply describe what happened and what they did. 
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Needs Improvement: 

- Sometimes encourages crew to analyze along CRM dimensions the situation that confronted them and what they did to 
manage the situation but does not push crew to discuss why they did what they did. 

- Verbally requests but does not pursue getting the crew to evaluate their performance and/or ways they might improve. 

- Encourages crew to explore CRM issues but does not ask crew to discuss how they specifically affect LOFT performance 
and line operations. 

- Tends not to push crew to analyze issues, factors, and outcomes in depth. Often settles for letting the crew simply 
describe what happened and what they did. 

Marginal: 

- Only minimally encourages crew to analyze along CRM dimensions the situation that confronted them and/or what they 
did to manage it. Does not push crew to discuss why they did what they did. 

- Only occasionally encourages crew to evaluate their performance and/or ways they might improve. 

- Occasionally encourages crew to explore CRM issues, and does not encourage crew to discuss how they affect LOFT 
performance or line operations. 

- Only occasionally encourages crew’ to analyze issues, factors, and outcomes in depth. Content for crew to describe what 
happened and what they did. 

Poor: 

- Does not encourages crew to analyze along CRM dimensions the situation that confronted them, what they did to manage 

the situation, or why they did it. 

- Rarely encourages crew to evaluate their performance or ways they might improve. 

- Rarely encourages crew to explore CRM issues. 

- Rarely encourages crew to analyze issues, factors, and outcomes in depth. 
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IP Use of Videos 


Outstanding: 

- Consistently shows an appropriate number of videos of appropriate duration to illustrate/introduce topics. 

- Consistently uses video equipment efficiently: is able to find desired segment without wasting time and pauses the video 

if talk begins while playing. 

- Actively evokes and consistently pursues thorough crew discussion of each video segment or topic. 

- Consistently has a point to make and uses the video to make that point. 

Very Good: 

* Usually shows an appropriate number of videos of appropriate duration to illustrate/introduce topics. 

- Usually uses video equipment efficiently: is able to find desired segment without wasting much time and pauses the 

video if substantial talk begins while playing. 

- Works to get crew to discuss most of the video segments or topics in detail. 

- Usually has a point to make and uses the video to make that point. 

Good: 

- Generally shows an appropriate number of videos of appropriate duration to illustrate/introduce topics. 

- Tends to use video equipment efficiently: is generally able to find desired segment without wasting much time and 

generally pauses the video if substantial talk begins. 

- Encourages crew to discuss most video segments or topics and refrains from lecturing to crew or hindering their 

discussion. 

- Generally has a point to make and usually uses the video to make a point. 


Adequate: 

- On average shows an appropriate number of videos, usually of appropriate duration, to illustrate and introduce topics. 

- On average uses video equipment somewhat efficiently, finding desired segment without wasting too much time and 

generally pausing the video if substantial talk begins while playing. 

- Generally encourages crew* to discuss video segments or topics, but may also lecture to crew, thereby somewhat 
discouraging thorough crew discussion. 

- Generally has a point to make, but the point is not always clearly tied to the video. 


Needs Improvement: 

- Shows somewhat too few or too many videos. Sometimes shows very short and/or very long segments while trying to 
illustrate/introduce topics. 

- Tends to use video equipment inefficiently: tends to waste some time trying to find desired segments and is slow' to pause 

the video if substantial talk begins while playing. 

- Sometimes encourages crew' to discuss video segment or topic, but may lecture, interrupt crew discussion, and/or not 

consistently pursue crew discussion. 

- Sometimes has a predetermined point to make, and sometimes uses the video to make a point. 
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Marginal: 

- Clearly shows too few or too many videos, sometimes of much too long and/or short a duration. Many videos not used 

to illustrate/introduce topics. 

- Uses video equipment inefficiently, wasting significant time trying to find desired segments while rarely pausing the 

video if substantial talk begins while playing. 

* Tends not to discuss video segments, and when they are discussed tends to lecture to crew about what occurred, only 
minimally encouraging crew to participate in a discussion. 

- Only occasionally has a point to make or uses the video to make a point. 


Poor: 

- Shows way too few or too many videos which are often much too long and/or short. Does not use videos to 
illustrate/introduce topics. 

- Uses video equipment very inefficiently: wastes substantial time trying to find desired segments and fails to pause the 

video if substantial talk begins while playing. 

- Usually does not discuss video segments, and when discussed usually lectures to crew without encouraging (and often 

hindering) crew participation. 

- Rarely has a point to make or uses the video to make a point. 
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Crew Analysis and Evaluation 


Outstanding: 

- Consistently analyze along CRM dimensions the situation that confronted them, what they did to manage the situation, 

and why they did it. 

* Consistently evaluate their performance and ways they might improve. 

- Consistently explore CRM issues and how they affect LOFT performance and line operations. 

- Consistently analyze issues, factors, and outcomes in depth, going beyond simply describing what happened and what 
they did. 

Very Good: 

- Frequently analyze along CRM dimensions the situation that confronted them, what they did to manage the situation, and 

why they did it. 

- Frequently evaluate their performance and ways they might improve. 

- Often explore CRM issues and how they affect LOFT performance and line operations. 

- Frequently analyze issues, factors, and outcomes in depth, going beyond simply describing what happened and what they 
did. 

Good: 

- Generally analyze along CRM dimensions the situation that confronted them and what they did to manage the situation. 
Briefly discuss why they did what they did. 

- Generally evaluate their performance and ways they might improve. 

- Generally explore CRM issues and how they affect LOFT performance and/or line operations. 

- Generally analyze issues, factors, and outcomes in moderate depth, usually going beyond simply describing what 
happened and what they did. 

Adequate: 

- On average analyze along CRM dimensions the situation that confronted them and what they did to manage the situation. 
Briefly discuss why they did what they did. 

- On average evaluate their performance and/or ways they might improve. 

- On average explore CRM issues and how they affect LOFT performance and/or line operations. 

- Analyze some issues, factors, and outcomes in some depth, often going beyond simply describing what happened and 

what they did. 

Needs Improvement: 

- Only part of the time analyze along CRM dimensions the situation that confronted them, what they did to manage the 
situation, or why they did it. 

- Only sometimes evaluate their performance and ways they might improve. 

- Sometimes explore CRM issues but give little discussion of how they affect LOFT performance or line operations. 

- Analyze only a few issues, factors, and outcomes in any depth, sometimes going beyond simply describing what 

happened and what they did. 
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Marginal: 

- Occasionally analyze along CRM dimensions the situation that confronted them. Occasionally discuss what they did to 
manage the situation or why they did it. 

- Only occasionally evaluate their performance and do not discuss ways they might improve. 

- Only occasionally explore CRM issues and do not discuss how they affect LOFT performance and line operations. 

- Analyze issues, factors, and outcomes in very little depth, rarely going beyond simply describing what happened and what 

they did. 

Poor: 

- Do little to analyze along CRM dimensions the situation that confronted them, what they did to manage the situation, or 

why they did it. 

- Rarely evaluate their performance or ways they might improve. 

- Rarely explore CRM issues and how they affect LOFT performance and line operations. 

- Do not analyze issues, factors, and outcomes in depth; only briefly describe what happened. 
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Depth Of Crew Activity 


Outstanding: 

- Consistently go substantially beyond minimal responses to IP questions. 

- Consistently participate deeply and thoughtfully. 

- Continually initiate dialogue and pursue issues to completion rather than just responding to questions, and consistently 

interact with each other rather than only with the IP. 

- Behave in a consistently proactive rather than reactive manner, being actively involved rather than just passing through 

the training. 


Very Good: 

- Frequently go substantially beyond minimal responses to IP questions. 

- Usually participate deeply and thoughtfully. 

- Frequently initiate dialogue rather than just responding to questions, and often interact with each other rather than only 

with the IP. 

- Usually behave in a proactive rather than reactive manner, being actively involved rather than just passing through the 

training. 


Good: 

- Generally go well beyond minimal responses to IP questions. 

- Generally participate deeply and thoughtfully. 

- Tend to initiate dialogue rather than just responding to questions and generally interact with each other rather than only 

with the IP. 

- Generally behave in a proactive rather than reactive manner, being actively involved rather than just passing through the 

training. 

Adequate: 

- On average go somewhat beyond minimal responses to IP questions. 

- On average participate somewhat deeply and thoughtfully. 

- On average initiate dialogue rather than just responding to questions and interact with each other rather than only with the 

IP. 

- On average behave in a proactive rather than reactive manner, being actively involved rather than just passing through the 

training. 


Needs Improvement: 

- Tend to give slightly more than minimal responses to IP questions. 

- Sometimes participate deeply and thoughtfully. 

- Tend to just respond to questions rather than initiate dialogue. Tend to interact with the IP more than with each other. 

- Sometimes behave in a more reactive than proactive manner. 
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Marginal: 

- Frequently give only minimal responses to IP questions. 

- Only occasionally participates deeply or thoughtfully. 

- Tend to just respond to questions rather than initiate dialogue. Only occasionally interact with each other; tend to interact 

only with IP. 

- Behave in a generally reactive rather than proactive manner. 

Poor: 

- Consistently gives only minimal responses to IP questions. 

- Rarely participate deeply or thoughtfully. 

- Rarely initiate dialogue; usually just respond to IP. Rarely interact with each other. 

- Behave in a consistently reactive rather than proactive manner. Appear to just pass through the training rather than being 
actively involved. 
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Appendix E. Spearman Correlation Coefficients 

* - Signif. LE .05 ** - Sigmf. LE .01 ***- Signif. LE .001 (2-tailed) 
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-.1424 

-.0452 

.0163 

.4651** 

.2832 

.3569* 

FEWPERUT 

-.2466 

-.1805 

-.0382 

.1609 

.0046 

.2151 

CREWPERU 

-.1298 

.0552 

-.0052 

.5212** 

.4160* 

.4927** 


DURATION 

CRMPERF 

TECHPERF 

SI_INTRO 

SI_QUEST 

SI_ENCRG 
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CASIUTPH 

.0427 

-.1121 

-.2770 

-.1161 

.1925 

.3937* 

FOSIUTPH 

.0815 

-.0038 

-.1711 

-.0087 

.2200 

.4783** 

FES I UT PH 

-.0909 

.1636 

-.3801 

-.2868 

.0727 

.4510 

CREWS IUT 

.1095 

-.0342 

-.2893 

-.1216 

.2326 

.5146** 

CANALUTT 

-.1476 

.1638 

-.0313 

.1808 

.4922** 

.6006** 

FOANALUT 

-.0394 

.0016 

.1167 

.0247 

.3895* 

.4894** 

FEANALUT 

.1777 

.2336 

-.2002 

.1233 

.3645 

.3676 

CREWANUT 

-.0263 

.1598 

-.0224 

.0699 

.5590** 

.7086** 

CREWPAQP 

.1906 

-.1825 

-.4826** 

-.1131 

-.0720 

.1048 

FOPAQPH 

. 3335* 

-.1568 

-.2807 

.0997 

-.0364 

.1480 

FEPAQPH 

-.1864 

.3065 

-.0102 

-.4594 

.0621 

.1126 

CAPAQPH 

.0735 

-.2203 

-.4341* 

-.1646 

-.1361 

- .0379 

NONDQ-CA 

-.1144 

.0641 

.0003 

-.0144 

-.0658 

.0390 

NONDQ_FO 

-.0939 

-.0471 

.0329 

-.0577 

.0513 

.0876 

NONDQ_FE 

-.2023 

.6371* 

.0000 

-.0721 

.2575 

.5023 

NONDQ_NO 

.3770* 

.0467 

-.0227 

-.0299 

.0828 

.0491 


DURATION 

CRMPERF 

TECHPERF 

SI_INTRO 

SI_QUEST 

SI_ENCRG 


S I_QEC 

SI_VIDEO 

IPPART 

IPPOS 

IPNEGIMP 

.9164** 

.3614 

-.4036* 

.0401 

-.0948 

.4699* 

-.5776** 

-.0920 

-.0407 

-.0551 

.1281 

-.3417 

.1329 

-.1460 

-.1979 


IPNEG 

-.2323 

-.0944 

-.2766 

-.1609 

-.2192 

.7942** 

I PIMP 

.0588 

.0300 

-.2193 

-.1672 

-.1198 

.8565** 

IPNEUT 

.3165 

.3478* 

.5350** 

-.1390 

-.0179 

-.1339 

IPPERF 

.1208 

.0371 

.2545 

.0520 

.7291** 

.1576 

IPCRM 

.4533** 

.3859* 

.6864** 

.0486 

.2782 

-.3150 

IPTECH 

-.5236** 

-.4760** 

-.6398** 

.0637 

-.1579 

.2108 

IPMIXED 

-.3117 

-.2663 

-.1770 

-.0544 

.0230 

.3470* 

IPNS 

.2112 

.2538 

-.1706 

-.3976* 

-.3283 

-.0397 

IPCRM2 

.4469** 

.3648* 

.6551** 

.0977 

.2982 

-.2635 

IPTECH2 

-.5883** 

-.5246** 

-.6997** 

.0790 

-.1677 

.3398* 

IPWPERUT 

-.3072 

-.4411** 

.0853 

.8200** 

.2448 

-.2443 

DIRQPHR 

.4956** 

.4743** 

.2442 

-.0886 

-.0663 

.0767 

IPDQjCA 

.4159* 

.3781* 

.0835 

-.0231 

-.0579 

.1045 

IPDQ_FO 

.4971** 

.5308** 

.2816 

-.1241 

- .1704 

.1013 

IPDQ_FE 

.2551 

.5182 

.9048** 

.2785 

.3455 

.5577 

NONDQPHR 

.0848 

.1064 

.1720 

-.2333 

.0415 

.0841 

TOTQPHR 

.5150** 

.5120** 

.3819 

-.1382 

-.0659 

.0703 

INTERUPH 

-.0021 

.0739 

-.1735 

-.4436** 

-.0304 

.4258** 

INTERRUP 

-.1414 

-.1605 

-.1458 

.1207 

.0092 

.1507 

INTER_UN 

.1483 

.1082 

-.1044 

-.1133 

-.0534 

.0781 

VSEGPERH 

-.0538 

-.0498 

-.0142 

-.0713 

-.1088 

.1043 

AVSEGDUR 

-.2481 

-.1864 

.0560 

.1889 

-.1141 

.0327 

SC_ACTIV 

.5137** 

.6813** 

.2614 

-.8441** 

-.0264 

.1025 

SC_CONT 

.7487** 

. 8242** 

.3279 

-.6702** 

. 0565 

.0264 

CAPART 

.3631* 

.4907** 

.1150 

-.6180** 

-.1737 

.2339 

FOPART 

.2210 

.3880* 

.0499 

-.8275** 

-.0451 

.0223 

FEPART 

-.0137 

.2909 

.0238 

-.7671** 

.2091 

-.1521 

CREWPART 

.4007* 

.5741** 

.0457 

-.9998** 

-.1413 

.1482 

CREWPOS 

.4954** 

.4052* 

.0876 

-.1581 

.3549* 

-.3315* 

CREWNEIM 

.0430 

.0597 

-.2126 

-.0520 

-.1627 

.7532** 

CREWNEG 

-.0717 

-.0332 

-.3366 

-.0888 

-.2845 

.7322** 

CREWIMPR 

.2078 

.1729 

-.0315 

-.1093 

t 

o 

H-* 

.6022** 

CREWNEUT 

.1688 

.2116 

.2330 

-.1397 

.0418 

-.0771 


SI_CONT 

SI_QEC 

SI_VTDEO 

IPPART 

IPPOS 

IPNEGIMP 
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CREWPERF 

.2539 

.2632 

.2464 

.0800 

.2325 

.1183 

CREWCEM 

. 6310* * 

.5550** 

.6691** 

-.0367 

.2144 

-.2777 

CREWTECH 

- . 4985** 

-.4228* 

-.6131** 

-.0084 

-.1464 

.3302* 

CREWMIX 

-.1967 

-.1489 

.2032 

-.0925 

-.0351 

.0401 

CREWNS 

-.3631* 

-.3257 

-.7452** 

-.0038 

-.3572* 

.3686* 

CREWCRM2 

.6446** 

.5660** 

.7317** 

-.0260 

.2643 

-.3642* 

CREWTEC2 

-.5404** 

-.4560** 

-.4977** 

-.0166 

-.1219 

.2831 

CAWPERES 

.3596* 

.4064* 

.1363 

-.3094 

.1671 

-.1853 

FOWPERES 

.1587 

.1353 

.1428 

.0188 

.1993 

-.2362 

FEWPERES 

-.0183 

.0639 

.2515 

.0482 

-.0274 

-.3449 

CREWPERE 

.2876 

.3270 

.1069 

-.1876 

.1909 

-.1563 

CAWPERUT 

.4171* 

.4573** 

.2877 

-.3675* 

.0824 

-.1926 

FOWPERUT 

.2232 

.2909 

.3186 

-.2833 

.1362 

-.2122 

FEWPERUT 

-.0297 

.0000 

.1205 

-.2638 

.2603 

-.3079 

CREWPERU 

.3862* 

.4503** 

.3259 

-.3764* 

.1707 

-.2239 

CASIUTPH 

.1348 

.2854 

.0654 

-.6366** 

-.1033 

.3896* 

FOSIUTPH 

.0860 

.2697 

.0265 

-.8281** 

-.0040 

.1751 

FES I UT PH 

.1412 

.1909 

-.2143 

-.7808** 

.0455 

-.1106 

CREWS I UT 

.1510 

.3278 

.0383 

-.8718** 

-.1005 

.3051 

CANALUTT 

.3916* 

.5358** 

.1723 

-.5119** 

.1219 

.1005 

FOANALUT 

.4202* 

.4424** 

.1758 

-.4703** 

.1881 

-.0955 

FEANALUT 

.1461 

.2597 

.2635 

-.4531 

-.2369 

-.1478 

CREWANUT 

.5355** 

.6371** 

.1606 

-.7068** 

.1051 

-.0047 

CREWPAQP 

-.1514 

-.0290 

-.1895 

-.3358* 

-.3882* 

.4817* 

FOPAQPH 

-.0756 

.0058 

-.1203 

-.2486 

-.4140* 

.3691* 

FEPAQPH 

.0240 

.2198 

.3546 

.0576 

.4062 

.5645 

CAPAQPH 

-.1990 

-.1045 

-.2513 

-.1733 

-.2573 

.4151* 

NONDQ_CA 

.0527 

.0037 

.2257 

-.0877 

-.0766 

-.1307 

NONDQ_FO 

.0231 

.0076 

-.0214 

-.1894 

.0892 

-.1900 

NONDQ—FE 

.1889 

.3357 

-.0976 

-.8037** 

.0736 

-.2774 

nontx2_.no 

.0259 

.0592 

-.2385 

.0337 

-.1362 

.2710 


SI_CONT 

SI_QEC 

SI_VIDEO 

IPPART 

IPPOS 

IPNEGIMP 
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IPIMP .4395** 


IPNEUT 

-.1263 

-.0604 





IPPERF 

.0732 

.1660 

.4550** 




IPCRM 

-.3486* 

-.1656 

.2497 

.1764 



IPTECH 

.2897 

.1049 

-.3892* 

-.1794 

-.7359** 


IPMIXED 

.4055* 

.1320 

-.0745 

.1651 

- . 5774* * 

.2114 

IPNS 

-.1367 

.0274 

.0956 

-.3378* 

-.3054 

-.0883 

IPCRM2 

-.2945 

-.1405 

.2466 

.2199 

.9621** 

-.7851** 

IPTECH2 

. 4272** 

.1715 

-.3803* 

-.1254 

-.8594** 

.9402** 

IPWPERUT 

-.2118 

-.3080 

-.1835 

.1260 

.2371 

-.1639 

DIRQPHR 

-.0880 

.1356 

.2457 

.1434 

-.0436 

-.0249 

IPDQ_CA 

-.0630 

.1456 

.1282 

.1504 

-.1201 

.0768 

IPDQ-FO 

-.0355 

.1501 

.1986 

-.0379 

.1637 

-.1558 

IPDQ-FE 

.3506 

.4360 

.1187 

.3736 

-.0820 

.0959 

NONDQPHR 

.1600 

.1628 

-.1312 

-.0597 

-.0942 

.2405 

TOTQPHR 

-.0302 

.1692 

.1724 

.1029 

-.0537 

.0841 

INTERUPH 

.3257 

.4298** 

-.1852 

-.0743 

-.2117 

.2476 

INTERRUP 

.0506 

.1151 

-.2439 

-.1019 

-.0470 

-.0046 

INTER_UN 

-.0074 

.0921 

-.1480 

-.1501 

-.0310 

-.0788 

VSEGPERH 

.0507 

.2162 

.0905 

-.0326 

-.1260 

.2894 

AVSEGDUR 

.1583 

-.1726 

-.0639 

-.1172 

-.0763 

-.1026 

SC_ACTIV 

.0467 

.1567 

.1422 

.0242 

.2755 

-.3878* 

SC_CONT 

-.0499 

.0790 

.2158 

.1230 

.4037* 

-.5309** 

CAPART 

.1443 

.2902 

.0240 

-.0362 

.0443 

-.0162 

FOPART 

.0592 

.0677 

.1382 

.0076 

i 

o 

- .0313 

FEPART 

-.2618 

-.1054 

.1142 

.0182 

.1412 

-.3379 

CREWPART 

.1645 

.1670 

.1349 

-.0616 

-.0522 

-.0629 

CREWPOS 

-.2978 

-.3226 

.1063 

.2957 

.1132 

-.2555 

CREWNEIM 

.5743** 

.6183** 

-.1603 

.1062 

-.0697 

.1107 

CREWNEG 

.6063** 

.5263** 

-.1302 

-.0311 

-.2695 

.1575 

CREWIMPR 

.3514* 

.6740** 

-.0301 

.2173 

.1116 

.0363 

CREWNEUT 

.0030 

-.0853 

.2206 

.1072 

-.0032 

-.2108 

CREWPERF 

.1291 

.0171 

.1861 

.4086* 

.0802 

-.2408 

CREWCRM 

-.3434* 

-.1837 

.3301* 

.2506 

.7550** 

- .7112** 

CREWTECH 

.4456** 

.1990 

-.4269** 

-.2004 

-.6894** 

.8469** 

CREWMIX 

.1209 

.0252 

.1592 

.0579 

-.3808* 

.0852 

CREWNS 

.2893 

.2581 

-.4427** 

-.4304** 

-.3637* 

.4517** 

CREWCRM2 

-.4316** 

-.2435 

.4476** 

.3158 

.7509** 

-.7826** 

CREWTEC2 

.3902* 

.1923 

-.2852 

-.1345 

-.7706** 

.7046** 

CAWPERES 

-.1898 

-.2329 

.0284 

.0914 

.3749* 

-.4673** 

FOWPERES 

-.2904 

-.2100 

.0190 

.1654 

.2522 

-.3212 

FEWPERES 

-.1385 

-.2792 

-.0528 

.0732 

.0343 

-.1147 

CREWPERE 

-.1736 

-.2203 

-.0144 

.1294 

.3556* 

-.4511** 

CAWPERUT 

-.1825 

-.2214 

.1167 

.0388 

.4711** 

-.5919** 

FOWPERUT 

-.1642 

-.1838 

.1196 

.1408 

.1950 

-.3330* 

FEWPERUT 

-.1949 

-.2479 

.1858 

.3158 

.3959 

-.6009 

CREWPERU 

-.1943 

-.2226 

.1757 

.1725 

.4020* 

-.5442** 

CAS I UT PH 

.3112 

.4112* 

-.0579 

-.0578 

.0304 

.0227 

FOSIUTPH 

.1909 

.1946 

.0049 

.0172 

-.0755 

.1043 

FESIUTPH 

-.2524 

-.0240 

.2466 

-.0683 

.1503 

-.4566 

CREWSIUT 

.2899 

.3056 

.0082 

-.0568 

-.0630 

.0079 

CANALUTT 

.1006 

.0301 

.0460 

.1463 

.1254 

-.1739 

FOANALUT 

-.1163 

-.0149 

.0155 

.0914 

.3638* 

-.4239** 

FEANALUT 

-.3655 

.0528 

.0938 

-.1553 

-.0868 

-.0892 

CREWANUT 

-.0475 

.0216 

.0819 

.0735 

.2296 

-.3679* 

CREWPAQP 

.5110** 

.4420** 

-.2524 

-.4011* 

-.2781 

.2315 

FOPAQPH 

.4622** 

.3138 

-.0706 

-.3189 

-.2032 

.0537 

FEPAQPH 

.1892 

.4937 

-.0216 

.1868 

-.0287 

.0384 

CAPAQPH 

.4025* 

.3833* 

-.2920 

-.3244 

-.1971 

.2943 

NONDQ_CA 

-.1721 

-.0135 

.2349 

.0080 

.0505 

-.1806 

NONDQ_FO 

-.1098 

-.0767 

-.0011 

-.0309 

-.0619 

.0266 

NONDQJFE 

-.1702 

-.3078 

.1940 

-.0714 

.1083 

-.1316 

NONDQ_NO 

.3704* 

.0548 

-.2307 

-.0898 

-.2099 

.2261 


IPNEG 

IPIMP 

IPNEUT 

IPPERF 

IPCRM 

IPTECH 
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IPNS 

IPCRM2 

IPTECH2 

IPWPERUT 

DIRQPHR 

-.1320 

-.3955* 

.4839** 

.0483 

-.1527 

-.4022* 

-.1150 

-.5308** 

.2048 

-.8383** 

.3110 

-.0724 

-.1053 

-.0789 

-.1914 


IPDQ_CA 

-.1337 

.1482 

-.1473 

.0226 

-.1217 

.9393** 

IPDQ_FO 

-.3157 

.2046 

.0936 

-.2262 

-.1922 

.8039** 

IPDQ—FE 

.2182 

-.2182 

.0364 

.1636 

-.0137 

.8929** 

NONDQPKR 

.2117 

-.2320 

-.0490 

.2333 

-.3348* 

-.0853 

TOTQPHR 

-.0651 

.0085 

-.0470. 

.0236 

-.2784 

.8323** 

INTERUPH 

.1828 

.1166 

-.2362 

.2571 

-.4455** 

.1886 

INTERRUP 

.1444 

-.1206 

-.0039 

.0826 

.2730 

-.0434 

INTER_JJN 

.0721 

.0834 

.0104 

-.0265 

.0591 

.0306 

VSEGPERH 

-.1160 

.1997 

-.2259 

.1696 

-.2053 

-.0892 

AVSEGDUR 

.1682 

-.1189 

-.0347 

.0438 

.1527 

-.2248 

SC__ACTIV 

-.0751 

.2566 

.2384 

-.3834* 

-.5495** 

-.0005 

SC_CONT 

-.1208 

.1880 

.4032* 

-.5208** 

-.3985* 

.2264 

CAPART 

-.0852 

.1396 

-.0011 

- .0567 

-.5008** 

.2575 

FOPART 

.0128 

.2593 

-.0713 

-.0540 

-.6163** 

.0007 

FEPART 

-.1182 

.6455* 

-.0455 

-.1818 

-.5890 

-.4146 

CREWPART 

.0548 

.4037* 

-.1013 

-.0775 

-.8224** 

.0844 

CREWPOS 

-.0097 

.1653 

.1335 

-.2730 

-.0214 

.2161 

CREWNEIM 

.1506 

-.1754 

-.0361 

.1650 

-.0854 

.0673 

CREWNEG 

.4016* 

-.0005 

-.2212 

.2685 

-.1441 

.0793 

CREWIMPR 

-.1407 

-.1784 

.1138 

.0098 

-.1738 

.1094 

CREWNEUT 

.1253 

.0982 

.0338 

-.1251 

-.0099 

.3613* 

CREWPERF 

. 1364 

-.0906 

.1282 

-.1516 

.1759 

.3697* 

CREWCRM 

-.5138** 

-.0223 

.7012** 

- . 8040** 

.1719 

.1765 

CREWTECH 

.3043 

-.0330 

-.6964** 

.8428** 

-.2321 

-.1464 

CREWMIX 

.6835** 

-.0656 

-.2421 

.2794 

-.1159 

-.0233 

CREWNS 

-.0173 

.2175 

-.4008* 

.4073* 

-.1866 

-.1524 

CREWCRM2 

-.4270** 

-.0340 

.7229** 

-.8393** 

.1910 

.2341 

CREWTEC2 

.6020** 

-.0504 

-.6989** 

.8218** 

-.1901 

-.1454 

CAWPERES 

-.0462 

-.0077 

.3974* 

-.4277** 

.0782 

-.0207 

FOWPERES 

.1025 

-.1842 

.3211 

-.2528 

.3306* 

-.0323 

FEWPERES 

.0776 

-.0183 

-.1005 

-.1553 

.6284* 

-.0023 

CREWPERE 

.0866 

-.1152 

.4146* 

-.3739* 

.2045 

-.0503 

CAWPERUT 

-.0023 

-.0213 

.5040** 

-.5465** 

.0549 

-.0512 

FOWPERUT 

.1364 

-.1145 

.2585 

-.2514 

.1146 

.1092 

FEWPERUT 

- .1279 

.1370 

.2420 

-.5160 

.2729 

-.4005 

CREWPERU 

.1137 

-.0985 

.4534** 

- .4710** 

.0778 

.0692 

CAS I UT PH 

.0401 

.1305 

-.0304 

.0098 

-.5240** 

.0266 

FOSIUTPH 

.0974 

.1331 

-.1201 

.0841 

-.6328** 

-.0225 

FESIUTPH 

-.1273 

.6182* 

.0818 

-.2909 

-.6986* 

-.4966 

CREWSIUT 

.1340 

.2529 

-.1110 

.0185 

-.6909** 

-.0195 

CANALUTT 

.0683 

.1104 

.0992 

-.1648 

-.2805 

.3096 

FOANALUT 

-.1979 

.1203 

.3227 

-.4267** 

-.1776 

-.0723 

FEANALUT 

.0592 

.2323 

-.0820 

-.0501 

-.1602 

-.1096 

CREWANUT 

-.0271 

.2712 

.1981 

-.3562* 

-.4140* 

.1566 

CREWPAQP 

.1144 

.2621 

-.3015 

.2486 

-.4976** 

.0693 

FOPAQPH 

.1815 

.2507 

-.1826 

.0942 

-.3522* 

-.0899 

FEPAQPH 

.0621 

.2246 

-.0669 

.1243 

-.4465 

.4599 

CAPAQPH 

.0439 

.0596 

-.2226 

.2924 

-.3005 

.0843 

NONDQ_CA 

.1291 

.0712 

.0551 

-.1368 

.0440 

-.0204 

NONDQ-FO 

-.2275 

.3705* 

-.1630 

-.0709 

-.2660 

-.0180 

NONDQ_FE 

-.3357 

.5977 

-.0230 

-.1241 

-.8291** 

-.4032 

NONDQ_.NO 

.3969* 

-.2624 

-.0928 

.3077 

-.0043 

-.0843 


IPMIXED 

IPNS 

IPCRM2 

IPTECH2 

IPWPERUT 

DIRQPHR 
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IPDQ_FO 

IPDQ__FE 

NONDQPHR 

TOTQPHR 

INTERUPH 

.6965** 

.7000* 

-.0822 

.8065** 

.1526 

.6758* 

-.0429 

.7103** 

.2137 

.5182 

.7882** 

.3326 

.4033* 

.2721 

.1747 


INTERRUP 

-.0574 

.0392 

-.3158 

-.1675 

-.1740 

.5489** 

INTER_UN 

-.0085 

.0012 

-.1545 

-.1207 

-.0769 

.4477** 

VSEGPERH 

-.1754 

-.0553 

.1905 

.0223 

-.1664 

-.0990 

AVSEGDUR 

-.1589 

-.3120 

-.4048 

.0580 

-.1140 

-.0009 

SC_ACTIV 

-.0505 

.1382 

-.3052 

.0945 

.0226 

.3627* 

SC_CONT 

.1390 

.3184 

.1327 

.0543 

.2088 

.2207 

CAPART 

.2603 

.2887 

.2096 

.0612 

.2807 

.3854* 

FOPART 

-.0130 

.1481 

-.3387 

.2553 

.1047 

.3439* 

FEPART 

-.4091 

-.3014 

-.1000 

-.1545 

-.3554 

-.2688 

CREWPART 

.0190 

.1232 

-.3098 

.2332 

.1337 

.4461** 

CREWPOS 

.1835 

.0875 

.2870 

-.0472 

.2252 

-.2244 

CREWNEIM 

.1062 

.0649 

.2648 

.1392 

.0643 

.3196 

CREWNEG 

.1130 

.0891 

.0412 

.1321 

.0476 

.5059** 

CREWIMPR 

.1148 

.0868 

.3494 

. 1930 

.1586 

.1754 

CREWNEUT 

.3054 

.1948 

.3273 

-.1349 

.1995 

.1746 

CREWPERF 

.3569* 

.1768 

.5740 

-.1032 

.2523 

-.0524 

CREWCRM 

.1049 

.3269 

-.3091 

-.2851 

.0848 

-.3296* 

CREWTECH 

-.0692 

-.1674 

.2700 

.3752* 

.0605 

.3402* 

CREWMIX 

-.0359 

-.2937 

.3059 

.3474* 

.0995 

.1304 

CREWNS 

-.0874 

-.1375 

-.2597 

-.1177 

-.2723 

.3067 

CREWCRM2 

.1440 

.3307* 

-.2091 

-.2543 

.1343 

-.3424* 

CREWTEC2 

-.0868 

-.3055 

.3091 

.4108* 

.0484 

.3145 

CAWPERES 

-.0904 

.1120 

-.2415 

-.3855* 

-.1958 

.0102 

FOWPERES 

.0128 

-.0950 

-.2727 

-.1095 

-.1053 

-.1297 

FEWPERES 

-.1096 

-.0734 

-.0548 

-.3607 

-.1190 

-.3959 

CREWPERE 

-.0751 

.0597 

-.1913 

-.2408 

-.1703 

-.0074 

CAWPERUT 

-.1298 

.0889 

-.2055 

-.3247 

-.1865 

-.0418 

FOWPERUT 

.0859 

.0815 

-.2014 

-.0636 

.0695 

.0593 

FEWPERUT 

-.4429 

-.3991 

-.2877 

-.7078* 

-.6110* 

-.3730 

CREWPERU 

.0121 

.1741 

-.1868 

-.1617 

.0113 

.0181 

CAS I UT PH 

-.0330 

.1082 

.2273 

.1537 

.0264 

.7105** 

FOSIUTPH 

-.0399 

.0873 

-.2455 

.2911 

.0864 

.6107** 

FES I UT PH 

-.4273 

-.2968 

-.3818 

-.2455 

-.4510 

-.1048 

CREWSIUT 

-.0781 

.0410 

-.3182 

.2012 

-.0106 

.6982** 

CANALUTT 

.2972 

.2892 

.2364 

-.0207 

.2105 

.4750** 

FOANALUT 

-.0647 

.0747 

-.5182 

.0869 

- .0507 

.1089 

FEANALUT 

-.2642 

- .1442 

-.0410 

-.1321 

-.0799 

-.6027* 

CREWANUT 

.1295 

.1897 

-.2460 

.0678 

.1004 

.3381* 

CREWPAQP 

.0207 

.1285 

.3781 

.2390 

.1102 

.5028** 

FOPAQPH 

-.0733 

-.0120 

.1535 

.3106 

.0572 

.1058 

FEPAQPH 

.5592 

.4441 

.5879 

.5448 

.4551 

.7712** 

CAPAQPH 

.0607 

.1576 

.3964 

.0933 

.0813 

.6262** 

NONDQ_CA 

-.0318 

-.0236 

-.2460 

-.0244 

-.0816 

-.0307 

NONDQ_FO 

.0554 

-.0543 

-.1150 

.1036 

.0222 

.0410 

NONDQ_FE 

-.3265 

-.0762 

-.2437 

.1058 

-.1429 

- .1313 

NGNDQ_NO 

-.0608 

-.1619 

.1169 

.3255 

.1145 

.1569 


IPDQ_CA 

IPDCL.FO 

IPDQ-FE 

NONDQPHR 

TOTQPHR 

INTERUPH 
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INTER_UN 

VSEGPERH 

AVSEGDUR 

SC_ACTIV 

SC__CONT 

. 8014** 
-.2676 
.2395 
. 1082 
.0924 

-.1850 

.2375 

.2726 

.2679 

-.7074** 

-.0797 

-.1552 

-.0789 

-.1150 

.8721** 


CAPART 

-.0580 

-.0335 

-.0108 

-.2827 

.6260** 

.5476** 

FOPART 

-.0394 

.0800 

.0895 

-.2717 

.7127** 

.5029** 

FEPART 

-.6133* 

-.4000 

.3571 

-.0476 

.5923 

.1785 

CREWPART 

-.1159 

.1164 

.0570 

-.1767 

.8434** 

.6696** 

CREWPOS 

-.2554 

-.1824 

-.1432 

-.1782 

.1422 

.3660* 

CREWNEIM 

.1363 

.1139 

-.0139 

.1367 

.1125 

.1332 

CREWNEG 

.3070 

.2169 

-.1471 

.2419 

.0835 

.0752 

CREWIMPR 

-.0601 

.0180 

.1798 

-.0492 

.1539 

.1793 

CREWNEUT 

.1775 

.2952 

-.2334 

.2762 

.1257 

.2030 

CREWPERF 

-.0168 

.0460 

-.1963 

.2906 

.0176 

.2342 

CREWCRM 

-.0435 

.0938 

-.0303 

-.1744 

.3440* 

.5645** 

CREWTECH 

-.0010 

-.1514 

.2469 

-.0627 

-.2718 

-.4695** 

CREWMIX 

-.0572 

.0134 

-.3908* 

.5376** 

-.0786 

-.1672 

CREWNS 

.2006 

.0991 

.0099 

.0089 

-.1148 

-.2475 

CREWCRM2 

-.0612 

.0874 

-.1198 

-.0889 

.3181 

.5539** 

CREWTEC2 

-.0070 

-.1106 

.0533 

.1903 

-.2868 

-.4932** 

CAWPERES 

.2194 

.2232 

-.1251 

-.3597 

.5312** 

.5860** 

FOWPERES 

.1552 

.1904 

.0432 

-.1932 

.1740 

.2191 

FEWPERES 

.3218 

.4703 

.6946 

-.5030 

.3364 

.2828 

CREWPERE 

.2365 

.2198 

-.0769 

-.3119 

.4472** 

.5037** 

CAWPERUT 

.1056 

.1748 

-.1736 

-.2671 

.5783** 

.5990** 

FOWPERUT 

.1680 

.2622 

-.1310 

-.1743 

.3940* 

.3836* 

FEWPERUT 

.0115 

.2466 

.8796** 

-.6506 

.5904 

.2276 

CREWPERU 

.1161 

.1681 

-.1923 

-.2460 

.5690** 

.5787** 

CASIUTPH 

.2080 

.1783 

.0469 

-.0321 

.6359** 

.4261** 

FOSIUTPH 

.1314 

.1444 

-.0409 

-.1349 

.6923** 

.4471** 

FES I UT PH 

-.5584 

-.5818 

.3333 

-.0476 

.6560* 

.3524 

CREWS IUT 

. 1439 

.1883 

.0041 

-.0496 

.7628** 

.5138** 

CANALUTT 

.1757 

.1688 

-.1895 

.0407 

.6313** 

.6498** 

FOANALUT 

.0978 

.3084 

-.3010 

.2060 

.6382** 

.5990** 

FEANALUT 

-.3601 

-.1185 

.4671 

- .1198 

.4886 

.4060 

CREWANUT 

.0731 

.2291 

-.2420 

.1195 

.8035** 

.8045** 

CREWPAQP 

-.0599 

-.0245 

.2674 

-.1392 

.1068 

-.1192 

FOPAQPH 

-.3475* 

-.1933 

.3556 

-.1745 

.0619 

-.1502 

FEPAQPH 

-.3032 

-.3776 

-.1909 

-.0273 

-.2922 

-.1227 

CAPAQPH 

.2510 

.1216 

.0721 

-.1327 

.0326 

-.1494 

NONDQ__CA 

-.1616 

-.1305 

.1983 

-.3035 

.0102 

.0021 

NONDQ—FO 

-.1619 

-.0294 

-.0322 

.0055 

.0931 

-.0482 

NONDQ_FE 

-.6644* 

-.7587** 

-.2684 

.2684 

.4263 

.1528 

NONDQ_NO 

.2396 

.2121 

-.1900 

.3059 

.0276 

.0508 


IOTERRUP 

INTERJJN 

VSEGPERH 

AVSEGDUR 

SC_ACTIV 

SC_CONT 
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FOPART 

.3255 






FEPART 

-.0223 

.6819* 





CREWPART 

. 6175** 

.8274** 

.7563** 




CREWPOS 

.1541 

.0063 

.1139 

.1535 



CREWNEIM 

.0034 

.0062 

-.7854** 

.0529 

-.2434 


CREWNEG 

-.0284 

.0652 

-.4623 

.0955 

-.3064 

.8433*’ 

CREWIMPR 

.0794 

.0510 

-.3632 

.1046 

-.1589 

.8195*’ 

CREWNEUT 

-.1238 

.1549 

.2182 

.1374 

.0137 

.0198 

CREWPERF 

-.2213 

-.1263 

-.0319 

-.0871 

.2930 

.3932* 

CREWCRM 

.0976 

.0121 

.1091 

.0323 

.3686* 

-.0665 

CREOTECH 

.0902 

-.0127 

-.2838 

.0107 

-.1637 

.1391 

CREWMIX 

-.1222 

.0279 

.0913 

.0919 

-.1653 

-.0859 

CREWNS 

.0341 

-.0528 

.0000 

.0139 

-.3955* 

.2562 

CREWCRM2 

.0365 

.0124 

.2091 

.0203 

.3909* 

-.1425 

CREWTEC2 

-.0055 

-.0161 

-.1455 

.0179 

-.2244 

.0368 

CAWPERES 

. 3525* 

.2718 

-.0228 

.3074 

.3896* 

-.1088 

FOWPERES 

-.2118 

.2142 

-.1091 

-.0232 

. 0705 

.0747 

FEWPERES 

-.1465 

.2046 

.0274 

-.0801 

.0000 

.0734 

CREWPERE 

.1102 

.3000 

-.0774 

.1855 

.2418 

.0634 

CAWPERUT 

.4544** 

.2247 

.2146 

.3662* 

.3275 

-.2040 

FOWPERUT 

.0939 

.4415** 

-.1236 

.2793 

.1550 

-.0184 

FEWPERUT 

-.1281 

.3793 

.4429 

.2265 

-.0435 

-.2890 

CREWPERU 

.3031 

.4420** 

.1230 

.3730* 

.3090 

-.1214 

CAS I UT PH 

.7339** 

.3639* 

-.0727 

.6380** 

-.1266 

.2069 

FOSIUTPH 

.4726** 

.8953** 

.4000 

.8284** 

-.0610 

.1006 

FESIUTPH 

.3007 

.5401 

.7909** 

-7882** 

.1503 

-.6667* 

CREWS IUT 

.6350** 

. 6784** 

.3545 

.8730** 

-.0416 

.1731 

CANALUTT 

.5414** 

.3437* 

.1727 

.5082** 

.2672 

.2190 

FOANALUT 

.0162 

.5989** 

.3182 

. 4700** 

.0817 

.1647 

FEANALUT 

.1826 

.2156 

.5421 

.4384 

.2283 

-.3936 

CREWANUT 

.3919* 

.5782** 

.5148 

.7048** 

.3030 

.1725 

CREWPAQP 

.3276 

.1419 

.2369 

.3396* 

-.3869* 

.1669 

FOPAQPH 

.0912 

.1993 

.5397 

.2512 

-.2738 

.2214 

FEPAQPH 

.0719 

-.0048 

.1434 

-.0575 

.0024 

.1032 

CAPAQPH 

.3555* 

.0349 

- .2192 

.1782 

-.3845* 

.0684 

NONDQ_CA 

.0156 

.1164 

.0182 

.0824 

.0364 

-.0861 

NONDQ_FO 

-.0820 

.3578* 

.6069* 

.1909 

-.0781 

-.2676 

NONDQ_FE 

.2742 

.6412* 

.6299* 

.8341** 

.3433 

-.8430* 

NONDQ_NO 

-.0800 

-.1155 

-.2150 

-.0312 

.0158 

.4074* 


CAPART 

FOPART 

FEPART 

CREWPART 

CREWPOS 

CREWNEIM 
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CREWIMPR 

.4525** 






CREWNEUT 

.0920 

-.0034 





CREWPERF 

.2573 

.3405* 

.6928** 




CREWCRM 

-.2761 

.0930 

.0133 

.2389 



CREWTECH 

.1986 

.0660 

-.2862 

-.2879 

-.7663** 


CREWMIX 

.1354 

-.1806 

.3615* 

.1431 

-.5470** 

.1328 

CREWNS 

.3938* 

.0780 

-.2160 

-.3571* 

-.5080** 

.3962* 

CREWCRM2 

-.3159 

.0181 

.1339 

.3166 

.9637** 

-.8524** 

CREWTEC2 

.1955 

-.0665 

-.0144 

-.1558 

-.9191** 

.8407** 

CAWPERES 

-.1432 

-.1435 

-.0413 

-.0362 

.4524** 

-.4058* 

FOWPERES 

.0000 

-.0111 

.2657 

.2887 

.2003 

-.4171* 

FEWPERES 

-.1333 

.0300 

.6347* 

.5606 

.3196 

-.2874 

CREWPERE 

.0296 

-.0659 

.1153 

.1541 

.3461* 

-.4188* 

CAWPERUT 

-.1893 

-.2439 

.0172 

-.0827 

.5069** 

-.5256** 

FOWPERUT 

-.0444 

-.0855 

.5080** 

.3274 

.2085 

-.3552* 

FEWPERUT 

-.3034 

-.1640 

.4338 

.2265 

.4977 

-.7655** 

CREWPERU 

-.0992 

-.1938 

.3139 

.1777 

.4087* 

-.4951** 

CAS I UT PH 

.2600 

.1634 

.1051 

-.1070 

-.0966 

.1821 

FOSIUTPH 

.1781 

.1182 

.1292 

-.1628 

-.1221 

.1704 

FESIUTPH 

-.3844 

-.3265 

.0091 

-.1913 

.2364 

-.3341 

CREWSIUT 

.2564 

.1454 

.1904 

-.0842 

-.1183 

.1497 

CANALUTT 

.2187 

.1127 

.4015* 

.4444** 

.1691 

-.1179 

FCANALUT 

.0722 

.2119 

.2646 

.2536 

.3994* 

-.4353** 

FEANALUT 

-.5138 

.1613 

.0911 

.1963 

.1913 

-.0642 

CREWANUT 

.1444 

.1754 

.3957* 

.3909* 

.3037 

-.3204 

CREWPAQP 

.2777 

.0720 

.0184 

-.2349 

-.4420** 

.4059* 

FOPAQPH 

.2768 

.1393 

.0295 

-.1080 

-.3247 

.2516 

FEPAQPH 

.2960 

.0387 

.0812 

.1485 

-.3393 

.1059 

CAPAQPH 

.2531 

-.0825 

-.0182 

-.3302* 

-.4381** 

.4343** 

NONDQ-CA 

-.0677 

-.0230 

-.1597 

-.1131 

.0859 

-.2083 

NONDQ_FO 

-.1979 

-.1318 

-.0403 

-.2170 

-.0811 

-.0183 

NONDQ__FE 

-.5116 

-.4163 

-.1333 

-.4608 

.0782 

.1782 

NONDQ_NO 

.4496** 

.2232 

.0068 

.1625 

-.2793 

.3743* 


CREWNEG 

CREWIMPR 

CREWNEUT 

CREWPERF 

CREWCRM 

CREWTECH 
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CREWNS 

CREWCRM2 

CREWTEC2 

CAWPERES 

FOWPERES 

-.1818 

-.3582* 

.6201** 

-.3014 

.0687 

-.6272** 
.2307 
- .1795 
-.3072 

-.8805** 

.4477** 

.2841 

-.4447** 

-.2644 

.4144* 


FEWPERES 

-.3211 

-.3227 

.2329 

-.3196 

.3570 

.7489** 

CREWPERE 

-.1563 

-.2041 

.3762* 

-.3872* 

.8480** 

.7899** 

CAWPERUT 

-.1737 

-.2041 

.5203** 

-.4900** 

.8735** 

.3000 

FOWPERUT 

. 1697 

-.4285** 

.2988 

-.1594 

.5123** 

.7613** 

FEWPERUT 

-.2523 

-.4302 

.5708 

-.5525 

.5881 

.6210* 

CREWPERU 

.0220 

-.4185* 

.4775** 

-.3633* 

.7810** 

.6157** 

CAS I UT PH 

.0425 

.1557 

-.1445 

.1454 

.1370 

-.1717 

FOSIUTPH 

.0749 

.0789 

-.1431 

.1476 

.2505 

.0304 

FESIUTPH 

-.0411 

-.1048 

.3273 

-.2000 

.0683 

-.2818 

CREWS I UT 

.1412 

.1161 

-.1404 

.1646 

.2152 

-.0971 

CANALUTT 

.0204 

-.1672 

.1857 

-.1102 

.3701* 

.2277 

FOANALUT 

-.0517 

-.2862 

.4288** 

-.3829* 

.3310* 

.4301** 

FEANALUT 

-.0664 

-.2557 

.1822 

-.1367 

.0183 

-.0410 

CREWANUT 

.0495 

-.2219 

.3398* 

-.2571 

.3776* 

.2591 

CREWPAQP 

. 1798 

.3765* 

-.4867** 

.4156* 

-.2774 

-.3370* 

FOPAQPH 

.2110 

.1988 

-.3534* 

.2961 

-.3330* 

-.1627 

FEPAQPH 

.3145 

.1269 

-.2246 

.2915 

-.4407 

-.4588 

CAPAQPH 

.0559 

.4428** 

- .4932** 

.3926* 

-.1299 

-.2885 

NONDQ__CA 

.1978 

-.3828* 

.1735 

-.0355 

.1863 

.2070 

NONDQ_FO 

-.0074 

.0982 

-.0724 

-.0115 

-.1763 

-.0918 

NONDQ_FE 

-.0901 

.2097 

.0782 

-.0460 

-.1244 

-.5472 

NONDQ_NO 

.2194 

.2285 

-.3275 

.3317* 

-.0552 

-.0090 


CREWMIX 

CREWNS 

CREWCRM2 

CREWTEC2 

CAWPERES 

FOWPERES 


CREWPERE 

.7368** 






CAWPERUT 

.4381 

.7063** 





FOWPERUT 

.7218* 

.7192** 

.4735** 




FEWPERUT 

.6858* 

.7643** 

.6560* 

.6345* 



CREWPERU 

.6087* 

.8377** 

.8217** 

.8458** 

.7895** 


CASIUTPH 

-.0594 

.0372 

.2816 

.0924 

-.0046 

.2033 

FOSIUTPH 

-.0365 

.1993 

.2065 

.3442* 

.1096 

.3530* 

FESIUTPH 

-.1872 

-.1913 

.2648 

-.2746 

.2831 

.0501 

CREWS IUT 

-.0137 

.1073 

.3054 

.2351 

.2146 

.3018 

CANALUTT 

.1187 

.3895* 

.3482* 

.3942* 

.1096 

.4741* 

FOANALUT 

.2648 

.4293** 

.2822 

.4693** 

.4566 

.4557* 

FEANALUT 

.3638 

-.0753 

.2449 

-.1376 

.2494 

-.0365 

CREWANUT 

.2014 

.3763* 

.3896* 

.4001* 

.2700 

.4802* 

CREWPAQP 

-.4348 

-.2973 

-.1309 

-.1339 

-.1190 

-.1717 

FOPAQPH 

-.1122 

-.2273 

-.1310 

-.0217 

.2173 

-.0631 

FEPAQPH 

-.5497 

-.4527 

-.4705 

-.4692 

-.3433 

-.3736 

CAPAQPH 

-.5247 

-.1877 

-.0598 

-.1218 

-.3115 

-.1341 

NONDQ—CA 

-.2769 

.1373 

.1553 

.1743 

.1808 

.1770 

NONDQ_FO 

-.3995 

-.2019 

-.2266 

-.1606 

.0554 

-.1739 

NONDQ_FE 

-.4411 

-.4539 

.0485 

-.4074 

-.1940 

-.1797 

NONDQJSTO 

.3780 

.0693 

-.1094 

-.0776 

-.1385 

-.0717 


FEWPERES 

CREWPERE 

CAWPERUT 

FOWPERUT 

FEWPERUT 

CREWPERU 
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FOSIUTPH 

.5963** 






FESIUTPH 

.2455 

.3545 





CHEWS IUT 

.8774** 

.8464** 

.7000* 




CANALUTT 

.5834** 

.4245** 

.3727 

.5705** 



FOANALUT 

.1434 

.4445** 

.2000 

.3451* 

.3581* 


FEANALUT 

-.0911 

-.0456 

.4875 

.1777 

.1458 

.0319 

CREWANUT 

.4684** 

.5389** 

.6150* 

.6349** 

.8049** 

.7468* 

CREWPAQP 

.5562** 

.2674 

.1822 

.4825** 

.0457 

-.1936 

FOPAQPH 

.2074 

.1205 

.4001 

.2401 

-.0698 

-.0980 

FEPAQPH 

.2820 

.1051 

.0526 

-.0239 

.2198 

-.1816 

CAPAQPH 

. 6022** 

.2624 

-.1772 

.4296** 

.0423 

-.2631 

NONDQ—CA 

-.1254 

.0434 

.3645 

.0161 

-.0218 

.1483 

NONDQ_FO 

-.1009 

.2204 

.3724 

.0338 

-.0807 

.3146 

NONDQ_FE 

.1012 

.6575* 

.6851* 

.5472 

.1517 

.1379 

NONDQ_NO 

.0208 

-.0185 

-.3693 

.0053 

.1047 

-.1515 


CAS IUT PH 

FOSIUTPH 

FESIUTPH 

CHEWS IUT 

CANALUTT 

FOANALUT 


CREWANUT 

CREWPAQP 

FOPAQPH 

FEPAQPH 

CAPAQPH 

NONDQ_CA 

NONDQ_FO 

NONDQ_FE 

NONDQJSTO 

.4429 

-.3881 

-.0676 

-.4096 

-.7455** 

-.1461 

-.0207 

.2765 

.3397 

-.0540 

-.0662 

-.0862 

-.1564 

.1017 

.0985 

.4194 

-.0197 

.7407** 

.7329* 

.8281** 

-.0536 

.1169 

.1290 

-.0156 

.4329 

.3531* 

.0445 

.1758 

.2871 

.0743 

.7159* 

.2251 

.4230 

.1039 

-.4866 

-.1750 

-.0020 

-.1297 

.0003 


FEANALUT 

CREWANUT 

CREWPAQP 

FOPAQPH 

FEPAQPH 

CAPAQPH 

NONDQ_FO 

NONDQ_FE 

NONDQ_NO 

.1322 

-.1590 

-.4557** 

.4884 

-.3257 

-.2601 





NONDQ—CA 

NONDQ_FO 

NONDQ_FE 





* - Signif. LE .05 ** - Signif. LE .01 (2-tailed) 

" . " is printed if a coefficient cannot be computed 


85 



Form Approved 
OMBNo. 0704-0188 


REPORT DOCUMENTATION PAGE 

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, 
gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this 
collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for information Operations and Reports, 1215 Jefferson 
Davis Highway. Suite 1204, Arlington, VA 22202-4302. and to the Office of Management and Budget, Paperwork Reduction Project (0704-0188). Washington, DC 20503. 

1. AGENCY USE ONLY (Leave blank) I 2. REPORT DATE 1 3. REPORT TYPE AND DATES COVERED 


March 1997 I Technical Memorandum 


4. TITLE AND SUBTITLE 

5. FUNDING NUMBERS 

LOFT Debriefings: An Analysis of Instructor Techniques and Crew 
Participation 

505-64-13-48-01 

6. AUTHOR(S) 

R. Key Dismukes, Kimberly K. Jobe, Lori K. McDonnell 


7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 

8. PERFORMING ORGANIZATION 
REPORT NUMBER 

Ames Research Center 
Moffett Field, CA 94035-1000 

A-976212 

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 

10. SPONSORING/MONITORING 
AGENCY REPORT NUMBER 

National Aeronautics and Space Administration 

NASA TM- 11 0442 

Washington, DC 20546-0001 

DOT/FAA/AR-96/122 

11. SUPPLEMENTARY NOTES 


Point of Contact: Key Dismukes, Ames Research Center, MS 262- A/4, Moffett Field, CA 94035-1000 1 

(415) 604-0150 


12a. DISTRIBUTION/AVAILABILITY STATEMENT 

12b. DISTRIBUTION CODE 

Unclassified-Unlimited 
Subject Category-03 



13. ABSTRACT (Maximum 200 words ) 


This study analyzes techniques instructors use to facilitate crew analysis and evaluation of their LOFT 
performance. A rating instrument called the Debriefing Assessment Battery (DAB) was developed which enables 
raters to reliably assess instructor facilitation techniques and characterize crew participation. Thirty-six 
debriefing sessions conducted at five U.S . airlines were analyzed to determine the nature of instructor facilitation 
and crew participation. Ratings obtained using the DAB corresponded closely with descriptive measures of 
instructor and crew performance. The data provide empirical evidence that facilitation can be an effective tool 
for increasing the depth of crew participation and self-analysis of CRM performance. Instructor facilitation skill 
varied dramatically, suggesting a need for more concrete hands-on training in facilitation techniques. Crews were 
responsive but fell short of actively leading their own debriefings. Ways to improve debriefing effectiveness are 
suggested. 


14. SUBJECT TERMS 15. NUMBER OF PAGES 

LOFT Debriefing, LOS, Facilitation, CRM, Training ^2 

IS. PRICE CODE 

AFO 

17. SECURITY CLASSIFICATION 1 18. SECURITY CLASSIFICATION 1 1 9. SECURITY CLASSIFICAtToN 2o7 LIMITATION OF ABSTRACT 
OF REPORT OF THIS PAGE OF ABSTRACT 

Unclassified Unclassified 

Standard Form 298 (Rev. 2-89) 

Prescribed by ANSI Std. Z39-18 
298-102 


NSN 7540-01-280-5500 


























